1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

247 

STATE OF MINNESOTA DISTRICT COURT 

COUNTY OF RAMSEY SECOND JUDICIAL DISTRICT 


The State of Minnesota, 
by Hubert H. Humphrey, III, 
its attorney general, 
and 

Blue Cross and Blue Shield 
of Minnesota, 

Plaintiffs, 

vs. File No. Cl-94-8565 

Philip Morris Incorporated, R.J. 

Reynolds Tobacco Company, Brown 
& Williamson Tobacco Corporation, 

B.A.T. Industries P.L.C., Lorillard 
Tobacco Company, The American 
Tobacco Company, Liggett Group, Inc., 

The Council for Tobacco Research-U.S.A., 

Inc., and The Tobacco Institute, Inc., 

Defendants. 


DEPOSITION OF DONALD B. RUBIN, Ph.D. 
Volume II, Pages 247 - 474 


STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 






CONFIDENTIAL 


248 

1 (The following is the Deposition of DONALD 

2 B. RUBIN, Ph.D., taken pursuant to Notice of Taking 

3 Deposition, at the offices of Dorsey & Whitney, 

4 Attorneys at Law, Pillsbury Center South, 220 South 

5 Sixth Street, Minneapolis, Minnesota, on October 7, 

6 1997, commencing at approximately 8:57 o'clock a.m.) 

7 APPEARANCES: 


8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


On Behalf of the Plaintiffs: 

Thomas Hamlin 

Robins, Kaplan, Miller & Ciresi 
Attorneys at Law 
2800 LaSalle Plaza 
800 LaSalle Avenue 
Minneapolis, Minnesota 55402 

On behalf of R. J. Reynolds Tobacco Company: 
Peter J. Biersteker 
Jones, Day, Reavis & Poque 
Attorneys at Law 
1450 G Street, N.W. 

Washington, D.C. 20005-2088 


STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




CONFIDENTIAL 


249 


1 



EXAMINATION INDE 

X 

2 

WITNESS 

EXAMINED BY 

PAGE 

3 

Donald B 

. Rubin, Ph.D. Mr. Hamlin 

250 

4 





5 





6 





7 



EXHIBIT INDEX 


8 

EXHIBIT 

DESCRIPTION 

MARKED 

9 

Plfs 

T 



10 

Ex. 

3541 

Article, "The control role of the 

346 

11 



propensity score in observational 


12 



studies for causal effects" by 


13 



Rosenbaum and Rubin, Biometrika; 


14 



16 pages 


15 


3542 

Article, "Matching Using Estimated 

354 

16 



Propensity Scores: Relating Theory 


17 



to Practice" by Rubin and Thomas, 


18 



Biometrics; 16 pages 


19 


3543 

Article, "Projecting From Advance 

361 

20 



Data Using Propensity Modeling: 


21 



An Application to Income and 


22 



Tax Statistics" by Czajka, et al.. 


23 



Journal of Business & Economic 


24 



Statistics; 15 pages 


25 


3544 

Facsimile w/attachments, 

411 


P .0. 

BOX 

STIREWALT & ASSOCIATES 

18188, MINNEAPOLIS, MN 55418 1-800- 

553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 


CONFIDENTIAL 

250 

Subject: Minnesota Files, 

Raghunathan to Rubin; 14 pages 
3545 Doc. ent. Section A: Main Effects 412 
Model; 18 pages 


7 


9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 


CONFIDENTIAL 

251 

PROCEEDINGS 
(Witness previously sworn.) 

DONALD B. RUBIN, Ph.D., 


4 having been called as a witness and having been 


5 previously duly sworn, testified under oath as 


6 follows: 


7 


CONTINUED EXAMINATION 


8 BY MR. HAMLIN:. 

9 Q. Good morning. Professor Rubin. 

10 A. Good morning. 

11 Q. In your work in this case did you make any 

12 attempt to impute missing data into NMES? 

13 A. No. 

14 Q. In your work in this case did you make any 

15 attempt to impute missing data into the Behavioral 

16 Risk Surveillance System? 

17 A. No, I did not. 

18 Q. Now let's — 

19 A. At least not yet. For both of those. 

20 Q. — go back to your work. Yeah. 

21 You haven't been asked to do that yet? 

22 A. No, I have not. Well I have not been asked by 

23 anyone involved in this case. 

24 Q. Let's return to your report which has been 

25 marked as Exhibit 3540. Would you turn to page 2. 
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A. Okay. 

Q. Now in the second full paragraph you state that 
one should first estimate the effect of that alleged 
lie on the smoking behavior of the individuals who 
were or would have been recipients of health care 
funded by Plaintiff. Then you go on to say that this 
would have several components and you list two of 
them. Do you see that? 

A. Correct. 

Q. The first component that you state on page 2 is 
as follows: "One would estimate how many current 
smokers in 1965, who were or would have been 
recipients of health care funded by plaintiffs, would 
have stop smoking sooner had defendants told the 
truth and how much sooner they would have quit." Now 
could you make that estimate from the NMES database? 
A. I don't believe so. 

Q. Could you — 

A. I don't believe so. 

Q. Could you make that estimate using the 
Behavioral Risk Factor Surveillance System database? 
A. Not by itself probably. I guess I should have 
said that about NMES as well, that I don't believe so 
by itself. It may be the data in either or both 
might be helpful in combination with other data from 
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other sources. 

Q. Okay. As you sit here today can you identify 
those other sources? 

A. Not as data sets that are off the shelf, but in 
the type of information that would be desired I can 
make suggestions, as we discussed yesterday. And 
thinking about that since then, I could also say that 
presumably there's some information around about 
purchasing of tobacco products after labels changed 
on them after the Surgeon General's report and what 
effect that had on people's behavior, that would be 
somewhat relevant to — to try to figure out how — 
how many or types of people would have stopped 
smoking or how long they would have continued to 
smoke, if — if there had been the — if the 
Defendants had not had this alleged misconduct at a 
particular date. 

Q. Can you identify a database containing that 
information? 

A. Excuse me. Not an off-the-shelf database that 
— that's available, but I would certainly think 
that people who know — knew about such databases and 
who know could access data that suggests what 
proportion of people smoked before and after label 
changes on the — on cigarette cartons, packs. 
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Q. But you don't know of an off-the-shelf database 
as you sit here today; right? 

A. As I sit here today I cannot name a particular 
one. 

Q. Now would you be able to estimate this effect 
that you described using the Medicaid claims 
database? 

A. Again not by itself. 

Q. I take it your answer would be the same for the 
Blue Cross Blue Shield database? 

A. Correct. 

Q. Now the second component that you describe is as 
follows: "One would also estimate how many fewer of 

the individuals who were or would have been 
recipients of health care funded by plaintiffs would 
have started smoking after 1965 had defendants told 
the truth." 

Now could you use the NMES database to estimate 
that effect? 

A. Again, not by itself, but perhaps in conjunction 
with other databases it could be helpful. 

Q. Could you use the Behavioral Risk Surveillance 
System database to estimate that effect? 

A. The same answer applies there, not by itself, 
but useful with other databases. For somebody who 
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knew more about how far back those kinds of surveys 
go and if you're talking about a system that's been 
going on, I don't know how long it's been in place, 
but if there's a collection of those surveys 
collecting smoking information frequently and going 
back 30 or 40 years, then perhaps that system of 
databases would be very helpful. 

Q. Now that system of databases that you are 
talking about were all compiled in the real world and 
not in some hypothetical world; right? 

A. All databases are compiled in — in the real 
world. 

Q. And let's assume that for the purposes of this 
question that the databases were compiled in a real 
world where the tobacco companies engaged in 
misconduct. Okay? 

A. All right. 

Q. And are you saying then that you can use those 
databases to determine what would have happened if 
the tobacco companies had not engaged in misconduct? 
A. I can use them in conjunction with — with other 
— I can use — start again. 

There's information in databases before the 
alleged mis — misconduct about behavior before the 
alleged misconduct and there's information in the 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

256 

databases after the alleged mis — misconduct about 
the world that exists with that alleged misconduct. 
Now that situation where you have to make assumptions 
and compare worlds with and without a treatment is 
what we always do when we estimate the causal effects 
of some treatment versus control conditions. That's 
just the way we operate when we approve drugs for 
distribution, when the FDA approves drugs that we do 
randomized experiments on the effect of — of school 
choice or any kind of treatment, effect of 
seatbelts. I mean that's the way we — we — we 
live, that's the way we — we draw conclusions. 
There's nothing so miraculous about doing that. 

Under certain assumptions we — we make decisions. 

Q. So I take it that the answer to my question is 
yes; right? 

A. I've talked so long I forgot what the question 
was. Maybe I should — 

Q. Well, here, let me — 

A. Maybe you should ask another question. I'll try 
not to talk so long this time. 

Q. Well, as you said, these databases which you 
can't identify as you sit here today were all 
compiled in the real world; correct? 

A. Correct. 
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Q. And your testimony is that you can use these 
real world databases to construct a hypothetical 
world where tobacco companies engaged in different 
kinds of conduct; right? 

A. I can — I can use them to help address 
questions about what would have happened in that 
other world with assumptions that would be — would 
be made explicit. 

Q. Now let's start — go back to your report on 
page 2. 

A. Okay. 

Q. The last sentence on page 2 states, "This kind 
of model would be based, in part, on data concerning 
the effect of" — "on smoking cessation and smoking 
initiation of the availability of additional 
information about smoking's health risks." 

Did I read that correctly? 

A. Yes. 

Q. Now you say that this model would be based in 
part on this data. What other data are you 
contemplating in this sentence? 

A. Well this sentence is about — specifically 
about smoking's health risk. I — I think other 
relevant data would concern general behavioral 
changes that occur when people are provided 
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information about other kinds of products' health 
risks, whether it's smoking or alcohol or skydiving 
or whether it's anything that's — that's risky I 
think you might find that there is a — a 
relationship between types of people and how they 
react to additional information about risky 
behavior. 

Q. You identified data concerning the effect on 
smoking cessation and smoking initiation. Do you 
have in mind a specific database or databases? 

A. Not a specific one, but as I mentioned before, I 
suspect there — there are databases that concern 
frequencies of smoking going back longitudinally to 
before there were labels on — on cigarette cartons 
and after. There may even be some experiments, I 
don't know, in psychology or sociology or economics 
where they actually try to do a little more 
controlled experiment to see what kind of behavior 
changes result from information like this. 

Q. Now in this model that you're proposing do you 
take into account the effect of addiction? 

A. I would certainly have to, because if some 
proportion of the population is addicted to tobacco, 
then I think what you would find is that the 
information that it's risky wouldn't have much effect 
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on their behavior, so there's some people who 
wouldn't be affected, and so if they were addicted 
before the alleged misconduct, the fact that they're 
addicted afterwards cannot — and their behavior 
doesn't change, that means there's no effect on them 
of the alleged misconduct. 

Q. Do you have a database in mind that you would 
make reference to to measure the effects of addiction 
to tobacco or cigarette smoking? 

A. I don't have a specific one, but since I — I 
know that's one of the issues in this case, I — I 
presume that the Plaintiffs could access databases or 
have databases suggesting the addictive powers of 
tobacco in what proportion, types of people can be 
addicted by it, so even though I cannot name a 
specific database I certainly expect that the data 
exist. 

Q. Have you examined the specific effects on 
behavior of addiction to tobacco? 

A. Have I? 

Q. Yeah. 

A. No, I have not. 

Q. But this is a component that you believe ought 
to be in your model? 

A. Yes. 
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Q. Let me change the — 

A. May I clarify that. Would be. I think it 
should be, and it would, sir, automatically be when 
you examine the effect of extra information on 
behavior, because you would find that there's some 
subgroup of people whose behavior doesn't change as a 
result of the information, and presumably people in 
that subgroup would be more likely, you'd have some 
proportion of them would be or might be addicted, and 
so it's automatic, you don't have to identify the 
addicted people, you don't care whether they're 
addicted or not in my model, but it's — you'll 
automatically find that there's a subgroup of people 
presumably whose behavior does not change or does not 
change much. And whether they have labels on them 
that they're addicted or not might not be important, 
but it might be important. 

Q. Now let me change this hypothetical world 
slightly. Suppose the tobacco companies came out 
with a safer cigarette in 1965 and people continued 
to smoke because the risks were less. 

A. Okay. 

Q. What database would you use to model that 
effect? 

A. Well, one starting assumption would be that the 
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only thing that would change would be the contents of 
the cigarette, so I think you're asking me to assume 
that behavior, smoking behavior doesn't change, but 
the materials inside cigarettes do change, and that's 
— in that case we would move on to compare the 
effect of the quote, "safer cigarette," just the 
health effects and health expenditures of the safer 
cigarette following 1965, if that's the date of the 
alleged misconduct, so we would — would measure that 
effect assuming the behavior is unchanged. I guess 
you're also assuming that only safer cigarettes are 
available or is the assumption that — that the old 
cigarettes that are bad for you and the new 
cigarettes that are better for you are both 
available? Because if we had both available then 
we'd have to — have to model what proportion of 
people would still go for the more dangerous 
cigarette even though a safer one's available. And I 
guess you have information on that because I — 
certainly some time in the past filtered cigarettes 
came out, and I think at the same time that 
unfiltered cigarettes came out, but I think people 
continued to buy the unfiltered cigarettes despite 
the fact that there's probably information, I don't 
remember, probably information that the filtered 
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cigarettes were supposed to be safer. 

Q. Do you have any expertise in making a 
determination about whether a filtered cigarette is 
safer from an unfiltered cigarettes? 

A. No, but I was using that not to determine 
whether it's safer or not but to address the question 
that you asked about the — this other world, and in 
that we'd have to model the behavior of people in 
choosing the safer cigarette if it were made 
available. For example, if both the old cigarettes 
and the safe cigarettes were available and everybody 
continued to — to smoke the old cigarettes, then the 
availability of the new cigarettes would have no 
effect. If everybody chose the new cigarettes, then 
it would have an effect. 

Q. Now assume for the purposes of the hypothetical 
that the only cigarette that's available is a safer 
cigarette. Would you use the databases on smoking 
cessation and smoking initiation to model that 
effect? 

A. It wouldn't be as critical, but I would — I'd 
have to think about that, but just a — as I'm 
sitting here I'm thinking about it. You might find 
— yeah, I think you would have to still consider 
those models because let's suppose only safer 
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cigarettes were available, they were advertised that 
way, I can certainly imagine that effect would be 
increased smoking, because there's some people who 
presumably don't smoke and never have smoked because 
of the — their awareness of the presumed health 
risks, and now if big banner headlines say "Safer 
cigarette comes out," I can imagine there are many 
people who wouldn't smoke in the world with the 
unsafe cigarette would smoke in the world with the 
safer cigarette, and in fact the health costs might 
be very different, and in fact might have 5 percent, 
10 percent, 20 percent more people smoking. 

Q. As you sit here today, can you identify a 
database which would give you that information? 

A. Well it's possible again that databases exist on 
the — in fact I'm sure that they do somewhere on 
sales of filtered and unfiltered cigarettes after 
filtered cigarettes came out. I think that would be 
relevant to address that question, what the relative 
sales were and how many more people or fewer smoked 
which kind of cigarettes. 

Q. Can you identify a specific database that has 
that information? 

A. I cannot identify a specific database. 

Q. Have you made an investigation to determine 
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1 whether such a database exists? 


2 

A. 

No, I have not. 



3 

Q. 

Now any database that 

you would look to 

would be 

4 

one 

that was compiled in a 

world where there 

was — 

5 

or there was not available 

a safer cigarette 

t 

6 

correct? 



7 

A. 

Well — 



8 


MR. BIERSTEKER: 

I'm sorry, I lost 

the 

9 

question. 



10 

Q. 

Well here — here — 



11 


MR. BIERSTEKER: 

Go ahead. 


12 

Q. 

Any database that you 

would look to would be a 

13 

database compiled in the real world? 


14 

A. 

Is that a question? 



15 

Q. 

Correct? 



16 

A. 

Yes . 



17 

Q. 

Yeah. And not in a world like the one 

that we 

18 

just 

set up where the only 

cigarette that's 

available 

19 

is a 

safer cigarette; correct? 


20 

A. 

As with all databases. 

they're collected in the 

21 

real 

world. 



22 

Q. 

Right. And looking at 

. those real world 



23 databases will tell you nothing about what would have 

24 happened if the world had been different; right? 

25 A. No. Wrong. 


STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

265 

Q. So you can look at that, at a real world 
database, and you can determine what would have had 
happened had the world been different? 

A. Not determine. You can estimate under certain 
assumptions. Otherwise we couldn't live our lives. 

We prepare for the weather tomorrow, we have never — 
we don't have tomorrow, we make — we make 
predictions about things. We don't — in a — We 
make decisions to wear seatbelts, do all kinds of 
things based on information we gather in the real 
world because we're comparing what would happen if we 
weren't wearing seatbelts. I mean that's what — 
what statistics does, that's what information does, 
that's how we make decisions. We make decisions to 
do things, things we're making implicit, sometimes 
explicit, comparisons of what the world would be like 
if we do something to what the world would be like if 
we didn't do something. That's the way we live. 

Q. Let me ask you this: How do databases which 
include information on smoking cessation tell you 
anything about what would happen in a world with a 
safer cigarette where no one quit? 

A. Well you want me to assume that — that — if 
you want me to assume that the behavior — let me 
start again. 
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If you want me to assume that in a world with a 
safer cigarette smoking behavior would be identical 
to what it is in a world — our world with the 
current cigarettes, then the answer is the effect of 
this alleged misconduct on behavior is nil, because 
you asked me to assume it's nil. Is that the 
question you asked? 

Q. No, I didn't ask you to assume that it was the 
same. 

MR. BIERSTEKER: I interpreted your 
question that way too, but go ahead. 

A. You can ask a different question then or 
rephrase it so I can answer. 

Q. Well my question was simply: What information 
could you use from a database showing smoking 
cessation to tell you anything about a world where 
there is a safer cigarette where you assume that no 
one quits smoking? 

A. I think it's the same question that I just 
answered. Because you're saying in a world where no 
one quits smoking you're asking me to assume that the 
smoking behavior has not changed. Is that right? 
Smoking behavior continues the same. If not, I don't 
understand the sentence. 

Q. Well what is it that you don't understand about 
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the sentence? 

A. I don't understand what it means other than the 
meaning I took it to mean and answered, and you're 
saying it has another meaning and I think you should 
— if you reformulate the question, I can try to 
answer. 

Q. Well let me ask you this: Do people quit 
smoking in the real world? 

A. Yes, I believe they do. 

Q. Or — 

A. I'm not an expert on that, but I know my father 
did it. Some people do. 

Q. And suppose you create a hypothetical world 
where no one quits smoking. 

A. Okay. 

Q. Okay. And those people in the real world who 
quit smoking are presumed to continue smoking. 

A. Okay. 

Q. Now those are two different worlds; right? 

A. I lost you there. I'm sorry. Maybe I shouldn't 
have, but I did. 

Q. Well you have the real world. 

A. Okay. 

Q. And in the real world some smokers quit 
smoking. 
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A. Correct. 

Q. Do you agree? 

A. That's World 1. 

Q. World 1. World 2 is a world where no one quits 
smoking. And in World 2 there's a safer cigarette. 
Okay? 

A. Okay. 

Q. All right. What can a database compiled in 
World 1 concerning smoking cessation tell you about 
World — world 2? 

A. Okay. When — if you tell me what variables are 
in the database compiled in World 1, then I could 
maybe tell you something, but you said it's a 
database compiled to address smoking cessation. Well 
there are databases that are collected for various 
purposes, and what the purpose of the database 
originally was does not say, does not limit the uses 
to which that data can be put. Okay. 

Q. And — 

A. So you have to — in order to answer that 
question you have to tell me what's available in this 
database. I don't care why it was collected really, 

I want to know what's in it, then I can tell you. 

Q. Well you're the one that made the reference to 
the data concerning the effect on smoking cessation 
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in your report, so it's that database or that data 
that I am referring to, and my question is simply: 

In this new hypothetical where there's a safer 
cigarette and no one quits smoking, would you use 
that smoking cessation database to model the new 
world? 

A. I — 

MR. BIERSTEKER: I'm going to object to 

form. 

A. Again, I might very well use that database, 
although if — if I'm told that the world is such 
that no one quits smoking, then that part of that 
database that addresses smoking cessation only would 
not be a relevant variable, whether somebody stops 
smoking and when they stopped smoking, that variable 
would not be relevant any more, but the rest of the 
data very well could be, might have information on 
smoking habits of people, how intensely they smoked, 
maybe even health care costs, it might have other 
information that's relevant, so that's why I wanted 
you to tell me what's in this other database. 

Q. Well as I said, aren't you the one that wrote in 
your report that you would look to data concerning 
the effect on smoking cessation; is that right? 

A. That's what I wrote, but I — 
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Q. And as you said, you don't have any specific 
database in mind so you don't know what that database 
would contain; right? 

A. I don't have any specific database in mind; 
correct. 

Q. Right. So you don't know what that database 
would contain? 

A. Correct. 

Q. Now let me ask you this: Suppose there is a 
continuing fraud by the tobacco companies from 1954 
to the present day, and assume that the only 
databases that you have are databases compiled in 
that world where there is a continuing fraud. Are 
you with me? 

A. Uh-huh. Yes. Well, no, let me ask a question. 
You mean the only databases that exist are from 1954 
and on? Is that what you're saying? Is that what 
you're telling me? I wasn't sure. In that world — 
you used 54 as a date in that world — 

Q. Yes. 

A. — and you want me to assume that the only data 
that are available are from 1954 and on or — 

Q. Well I want you to assume that that's the real 
world. All right. And assume that the only data you 
have available to you is data from the real world, 
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whether it's after 1954 or before 1954. 

A. Okay. So I — I have access to data going back 
if I can find it. 

Q. Right. 

A. Okay. 

Q. Okay. Now what can that data compiled in the 
real world where there is this continuing fraud tell 
you about a world where there is no fraud? 

A. Well, as I've tried to outline in the report, 
there — if there were no fraud, something would have 
been different in 1954, and you would have to sort of 
tell me what that thing that would be different would 
be, and then the first part of my effort would be to 
try to find data on what — how people's smoking 
behavior would change as a result of this thing that 
you can tell me they did instead of the fraud, which 
we're — which we're seeing now, and that kind of 
data concerns how people's behavior — the first part 
of it at least, how people's behavior changes as a 
result of the kind of information that you can tell 
me, kind of new — non-fraudulent behavior of the — 
of the tobacco industry. And there — as we spoke 
before, there are studies, there are data sets that 
concern how people's behavior changes as a — as a 
function of information. And that's why I said, it's 
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— it's — I believe that information on how 
people's smoking behavior changed when labels went on 
cigarettes, warning labels went on cigarettes, I 
imagine that marketing, advertisements, in the 
industry of advertising and marketing has information 
on how people are influenced by advertising, what 
kind of people are. Certainly the fact is we — "we" 
meaning the world — we model things all the time 
with — in our world as we see it and make decisions 
about what effects those things are going to have, 
and what we're doing there is we're comparing the 
world with this and the world without it. Every time 
somebody decides to put an advertisement in the 
newspaper of a particular form they're making a 
decision as to how much income they'll get with the 
advertising or without the advertising, or they 
choose between two different advertising agencies, 
they're thinking about — about the world with this 
advertisement type 1 versus advertisement type 2, 
they choose advertisement type 1 and they collect 
data on how much their sales go up or down, and they 
maybe change things, they make a decision to change 
advertising companies because their sales went down. 
That's what we do. 

Q. Let me ask you this: If in 1954 the tobacco 
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companies had told the truth about their product, 
would you be able to take into account any government 
regulation that might have resulted from that 
disclosure in your hypothetical world? 

A. That would be — could be part of a model that 
would — that would suggest what — what changed, and 
it would be — there probably wouldn't very good data 
on what kind of government regulation might come 
about, but you could make different assumptions about 
what the government would have done, and with the 
data that you have collected, the combination of 
assumptions and — and data would lead to an answer. 
If someone doesn't like the assumptions, then you can 
change the assumptions, see how the answer would 
change. In many situations the answers can be 
relatively insensitive to changing assumptions like 
that. And that's something — that's one of the 
issues that would — that has to be addressed if the 
question of what's the effect of alleged misconduct 
is to be addressed. I'm not saying it's an easy 
question to address, but if that's the — if that's 
the question, then you have to work to answer that 
question and not answer some other irrelevant 
question. 

Q. Do you know of any data sets you would use to 
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answer the question about government regulation in 
the event that the tobacco companies have told the 
truth? 

A. Do I know a particular off-the-shelf data set? 
No. But perhaps there are relevant data regarding 
what kind of actions government — our government 
takes when it's found — when a drug that's being 
marketed is found to have negative side effects; for 
example, how — how long they — even birth control 
pills, they ratcheted down the estrogen levels, the 
hormone levels in birth control pills by dramatic 
percentages over the years. How long does that take 
to take place? I don't know. But certainly there 
are data that are available from the FDA on things 
like that. How long does it take for regulations to 
come in effect to change the composition of drugs 
when they're found to have some potential negative 
side effects. 

Q. Have you ever made use of any such data? 

A. Have I made use of it? No, I have not made use 
of it, but I've — well made use in the sense of — 
of having a variety of conversations with — with 
people who were quite knowledgeable about such 
things, I have, so made use in that sense, to talk to 
them about it, fairly lengthy conversations I — I 
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have. 

Q. But you yourself haven't used that data? 

A. I haven't taken that data set, no, or those 
kinds of data sets, no, I have not. 

Q. And you wouldn't be able to identify them as you 
sit here today, would you? 

A. If I could make a phone call, I could. 

Q. But you can't as you sit here today? 

A. No, not without making a phone call. 

Q. Now let me direct your attention to page 3 where 
you set up an example where in the left-hand column 
Defendants Lie, the right-hand column Defendants are 
Truthful. Do you see that? 

A. Yes. 

Q. And then you describe two scenarios. Now I take 
it one is what you would call a real world scenario; 
is that right? 

A. Well in the hypothetical example they're both — 
yeah, the — the left column is — is supposedly as 
the world is as we see it with the alleged lie. 

Q. Right. And the right-hand column is the 
hypothetical or make-believe world; right? 

A. That's right, that's the hypo — that's the 
hypothetical other world. 

Q. Now, to determine what would happen in this — 
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in the real world do you have in mind any databases 
that you would use? 

MR. BIERSTEKER: Object to the form. 

A. I don't think you asked the question properly. 
You asked to determine what would happen in the real 
world do I have any idea what databases I would use? 
And the databases in the real world are the ones that 
exist. 

Q. Right. And you would use those databases for 
both the real world and your hypothetical world; 
right? 

A. Correct, because they tell me what happened in 
the real world, and by combining data sets from the 
real world I would try to — I would address what 
would have happened in — in this hypothetical world. 
Q. Now after you describe this real world and — 
and hypothetical world, you conclude on page 4 that 
"The measure of any increased health care costs to 
plaintiffs due to the alleged misconduct by 
defendants plainly will be different in different 
years." Do you see that? This is on — 

A. Right. 

Q. — the top of page 4. 

A. I'm just trying to get exactly where that 
sentence appears. I know it is in the middle of the 
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1 page, right, below — 

2 Q. Right. 

3 A. Oh, fine. Right. 

4 Q. Now in that statement you are assuming that the 

5 — well, strike that. 

6 In your example the smoking history in the real 

7 world and in the hypothetical world are different; 

8 right? 

9 A. Correct. 

10 Q. In the — 

11 A. Because if they were not, then there would be no 

12 effect of the alleged lie. Everything would be the 

13 same, in which case there's no — no effect of the 

14 alleged lie on this person whatsoever. 

15 Q. In the real world where Defendants lie, the 

16 person starts smoking in 1955 and quits smoking in 


17 

1991 

; correct? 



18 

A. 

Correct. 



19 

Q. 

And In the 

hypothetical world this person start: 

20 

smoking in 1955 

and quits smoking in 

1965; right? 

21 

A. 

Correct. 

Because — 


22 

Q. 

All right. 



23 

A. 

Before the 

alleged lie, because 

the alleged lie 

24 

cannot have an 

effect on something before that 


25 happened. 
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Q. So their smoking histories are different? 

A. After the alleged lie — 

Q. All right. 

A. — the smoking histories are different. 

Q. And you conclude from that that therefore their 
health care costs are different; right? 

A. Might well be different. 

Q. And they will be different because those who 
smoke for a longer period of time will have more 
disease and therefore more health care costs; right? 
A. Not necessarily. 

Q. You're just saying that they would be different? 
A. I'm just saying they'd be different. I'm not — 
Q. You don't know which one would have higher 
health care costs? 

A. No, I don't. I don't know in which world they'd 
have higher health care costs. 

Q. Okay. 

A. That's correct, I don't know. 

Q. All right. Now if we continue on down page 4, 
this is on page 4, you say "In the absence of direct 
data concerning the effect of defendants' alleged 
misconduct on health care costs funded by plaintiffs, 
two models should be estimated from appropriate 
data." Do you see that? 
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A. Yes. 

Q. Then you describe these two models, "One for the 
effect of that alleged misconduct on the subsequent 
smoking behavior of individuals who were or would 
have been recipients of health care funded by 
plaintiffs; and," you say, "another for the effect on 
health care costs of those changes in smoking 
behavior." 

Do you see that? 

A. Yes. 

Q. Now again, I want to ask you what are you 
referring to when you say "appropriate data". Do you 
have any specific data sets in mind? 

A. I do not have specific off-the-shelf data sets 
in mind, although the kind of data sets we discussed 
yesterday and today are the kinds that I was alluding 
to by the word "appropriate". 

Q. When you say "the kinds of data sets," do you 
mean these — this criteria that you set out for data 
sets you would like to have? 

A. That — that was — yeah, that's what we talked 
about yesterday, and the — the data sets I talked 
about today which, for example, are the effect of 
smoking of labeling changes, putting labels, 
cautionary labels on — on cigarettes, other data 
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sets perhaps related to the information being 
available on risks of other kinds of behaviors; 
drinking, seatbelt use. The things — what — how — 
how people respond to information that — that 
behaviors — certain behaviors have more health risks 
associated with them. That's the first part. The 
second part would be the appropriate data sets would 
be on health — related to health care costs for 
smokers and nonsmokers. 

Q. Do you know of any data sets that include 
behavioral changes and health care expenditures? 

A. As I sit here I don't. Some may exist. But I 
would certainly ask somebody, people who know about 
that at least. As I said many times, I'm not an 
expert in — in data sets generally, available data 
sets generally, and I'm certainly not an expert on 
specific data sets for behavior and smoking, but I 
certainly have — have names of people whom I would 
call. 

Q. But you haven't called those people in this 
case; right? 

A. I have not. 

Q. Now let me direct you to the sentence that 
begins, "For instance..." 

A. Uh-huh. 
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Q. On page 4. 

A. Okay. 

Q. That sentence reads, "For instance, any 
increased health care costs due to smoking that 
occurred before any alleged misconduct cannot be 
attributed to that alleged misconduct." 

Do you see that? 

A. Yes. 

Q. Now let me ask you this: I want you to assume, 
and this is based on the Plaintiffs' complaint, that 
the tobacco companies engaged in a fraud beginning on 
— in 1954 and that that fraud continues through 
today. All right. 

A. All right. 

Q. Consider this: A smoker begins smoking before 
1954 and continues to the present day. That smoker 
is also on Medicaid and incurs health care costs 
after 1978. 

A. After 1978? 

Q. Right. 

A. Okay. 

Q. Which is the damage period that the Plaintiffs 
claim in this case. 

A. Okay. 

Q. All right. Under your theory here can the 
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Plaintiffs or the State of Minnesota recover for that 
person's health care costs? 

A. Potentially, yes. 

Q. Okay. Now let me ask you to turn to page 5. 

A. Just to clarify, because if in 1954, let's say, 
the — in this scenario, the fraud didn't exist, 
there's a certain probability — we try to model as a 
function of background characteristics of specific 
people — that this person who started smoking before 
1954 would have stopped smoking at some time after 
1954, the probability he would have stopped maybe in 
1954, the probability he would have stopped in 1955, 
1956, 57, so there's a — a distribution over what he 
would have been like in the world without the alleged 
fraud, and in that — in that — in each of those 
scenarios which is given — has a distribution, each 
of those scenarios there could be different health 
care costs, different smoking behavior and different 
health care costs than there are in the real world, 
and so consequently there could be different health 
care costs in 1978 than there — in that world, and 
that's what is the question. That's exactly what the 
question is. And that's — in order to address that 
kind of question, these are the kinds of issues that 
have to be addressed. 
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Q. Now, let me ask you to turn to page 5. And I 
want you to direct your attention to the first 
paragraph where you describe the Zeger models. You 
say "Instead..." 

A. Yes, I see that. 

Q. You see that. You go on to say, "Instead, these 
models simply attempt to estimate the difference in 
health expenditures, through 1996, between smokers 
(current and former) and never smokers of similar 
background characteristics, such as age and sex, 
regardless of when or why that smoking occurred." 

Now that — that is your description of the Zeger 
models; correct? 

A. Correct. 

Q. Now would you agree that the — that Zeger, et 
al. took their data from real world databases like 
NMES and Behavioral Risk Surveillance System? 

MR. BIERSTEKER: Objection; asked and 

answered. 

MR. HAMLIN: I don't think — I don't think 
that's right. 

MR. BIERSTEKER: It is, since he said all 
data came from the real world, but never mind. 

Q. No. The question is: Would you agree that 
Zeger et al. took their data from real world 
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databases like NMES and BRFSS, B-R-F-S-S, which is 
the acronym? 

A. That's what they assert although they did make 
up some data by imputation, but excluding that, their 
description is that they took data from real data 
sources. 

Q. And you yourself have taken data from real data 
sources and made up data as well; correct? 

A. Not that way. 

Q. But you have done it in work that you've done, 
you may have done it differently, but you did impute 
data; right? 

A. Not that way. That's like saying somebody — 
somebody drives a car completely drunk at 140 miles 
and hour and you say. Ah, you too have driven a car, 
haven't you? 

Q. Well let me ask you this: You certainly have 
dealt with databases where there were missing values; 
right? 

A. Yes. 

Q. And you have filled in those missing values; 
right? 

A. Not that way. 

Q. I'm asking you if you yourself filled in those 
missing values? 
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A. Singly. 

Q. Yes or no? 

A. Have I ever filled in and then gone ahead and — 
and ignored the fact they were filled in? No, I 
haven't done it that way. I haven't done it. No, 
the answer is — the implication is I've done what 
they've done, and I've never done what they've done 
and never would do what they've done. 

Q. I didn't ask you — I didn't ask you whether you 
did what they did. I only asked you simply whether 
or not you have ever imputed missing values into a 
database? 

A. And by "imputed" you mean according to some 
method or — or just any kind of method? Imputed is 
whatever kind of method — okay, I'm going to be — 
try to be clear because I'm not saying it right. 
They've imputed values in their data sets, I have 
done imputations in data sets as well, but the 
methods are different. Okay. 

Q. That's all — that was my point. 

A. Fine. 

Q. I mean you both have done imputations, but as 
you say, you've done it differently; right? 

A. Validly versus invalidly. Driving at 130 miles 
an hour while drunk versus driving safely. That's 
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the — that's the difference. We both drove. 

Q. All right. For example, you have used multiple 
imputation to impute missing values into databases; 
right? 

A. Correct. 

Q. Now, would you agree that in real world 
databases like NMES and Behavioral Risk Factors 
Surveillance System the effect of the actions of the 
tobacco companies in the real world is reflected in 
that data? 

A. Yes. 

Q. Now, let me turn to page 6 of your report, 
specifically to the first full paragraph. In that 
first sentence you state, "Apart from any applicable 
legal considerations, the question of the effect of 
defendants' alleged misconduct on the health care 
expenditures of the plaintiffs in principle could be 
addressed by statistical analyses of appropriate 
data." 

Do you see that? 

A. Yes, I do. 

Q. Are you saying that there are statistical 
methods for determining the excess medical care costs 
incurred by smokers? 

A. Not determining, but estimating. 
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Q. Oh. 

A. In combination with assumptions that are 
explicitly stated. Yes, there are. 

Q. Now you have not performed that exercise; right? 
A. In — for tobacco? 

Q. Yes. 

A. Or that exercise in other data sets? 

Q. No, for — you have not created a — a model to 
estimate the excess medical care costs of smokers? 

A. No, I have not. 

Q. Okay. And I — what do you mean by the term "in 
principle" in that sentence? 

A. Well at that time when I wrote this I — I 
hadn't spent as much time thinking about what I would 
actually try to do, and so it was a — it was a — 

"in principle" was a phrase that was inserted to 
say, well, in principle you can do this because I've 
done it in observational studies in other settings; 
for example, like the breast cancer one we spoke of 
yesterday was an observational study, so in principle 
I know it can be done, but had I — I hadn't thought 
about the details of where you would actually get the 
kind of data, I hadn't thought about the analogies 
between this setup with tobacco and medical ex — 
expenditures and other examples I've worked on that 
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1 are — that are simpler, so I hadn't thought about 

2 that as much as I — I have since then, and so it was 

3 a statement that in principle, but if at the time 

4 someone said. Okay, start talking about the things 

5 we've talked about today and yesterday, for example, 

6 I would not have been as responsive I think as I've 

7 tried to be yesterday and today. 

8 Q. Let's assume that you are asked to design a 

9 model to estimate the excess medical care costs of 


10 

smokers versus 

those who do not smoke. 



11 

A. 

To clarify 

— may I ask a question 

to clarify? 

12 


MR. BIERSTEKER: I was going 

to object 

to 

13 

form so you may 

as well. 



14 

Q. 

I haven't 

asked a question yet. I 

haven' t 

asked 

15 

a 

question. 




16 

A. 

You said a 

hypothetical and I just 

want to 


17 

clarify what I' 

m supposed to assume so 

far. Because 

18 

you said smokers not — not smoking and 

this is 

for 

19 

alleged misconduct or fraud beginning in 54 let' 

s say 

20 

to 

be specific. 




21 

Q. 

No. Here' 

s my hypothetical. 



22 

A. 

Okay. 




23 

Q. 

Assume — 

assume that you are asked to create a 

24 

model to estimate the excess medical care costs 

of 

25 

smokers versus 

those who do not smoke. 

That's all 
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you're asked to do. All right. 

A. I don't know what it means. 

Q. Okay. So what — what you're — my first 
question is: What would you do? 

A. Did — 

MR. BIERSTEKER: Well, I I'll object to 

form. 

A. I would first — the first thing I would do, I 

— I would ask the framer of the question what he 
meant by" excess. 

Q. Do you have an understanding of what the term 
"excess" means? 

A. I know what the English word means, but I don't 

— I don't know what it means in the context of the 
sentence. 

Q. What does — what does it mean? 

A. It can mean different things. I'll give you two 
examples. 

MR. BIERSTEKER: I have a suggestion that 
might help. I wont' say it if you don't want me to. 
MR. HAMLIN: Peter. 

MR. BIERSTEKER: All right. Okay. Leave 
is granted. 

MR. HAMLIN: I'll always listen your 
suggestions. 
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1 MR. BIERSTEKER: Leave is granted. I think 

2 there's two ways to construe excess medical care 

3 costs of smokers compared to nonsmokers. One way 

4 would be to what the health care costs of smokers 

5 compared to what they would have been had they not 

6 smoked. That's one way to construe that question. 

7 The other way to construe the question is to compare 

8 the health care costs of smokers to the health care 

9 costs of nonsmokers as they both exist in the world 


10 

today. And I 

think that may be 

a source of 


11 

confusion. 




12 

A. That's what I was going to 

ask to each of those 

13 

two. 




14 

MR. 

BIERSTEKER: So if you want to 

— 

15 

anyway, go ahead. 



16 

MR. 

HAMLIN: Okay. 



17 

BY MR. HAMLIN 

: 



18 

Q. Let's try for clear up any 

confusion. Assume 

19 

that all you' 

re asked to do is 

determine the 

excess 

20 

health care costs of smokers versus those who 

do not 

21 

smoke in the 

real world. 



22 

A. So if — 

if you're saying 

compare the 19 

— some 

23 

— some year. 

1996, the amount. 

total amount 

of 

24 

health care - 

- I guess the average per person 

health 

25 

care costs in 

this group of people who are smokers 
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and this other group of people who — over here who 
are nonsmokers, including infants, just everybody. 

Is that right? I guess that's the question. So I 
add up all the health care costs for people who are 
smokers and divide by the number of them, and then I 
would add up all the health care costs for those 
people who are nonsmokers and divide by the number of 
them. And I'd say, oh, I have — I have two numbers, 
and I'll — I'll look at them, maybe I'll take the 
difference or maybe I'll take the ratio or something 
like that. I'll compare them. And if that's what — 
if that's what's meant by comparing health care costs 
for smokers and nonsmokers, then it's a simple 
descriptive exercise. And if you want to call the 
difference excess, maybe — maybe it's an odd use of 
"excess," maybe it's not an odd use of the word 
"excess". But certainly if — if the use of the 
word "excess" there is supposed to impiy that the 
difference reflects what the health care costs of 
smokers would be if they didn't smoke, that 
calculation has nothing to say about that. 

Q. All right. Let's — let's do it this way: 

Let's go back to page 5. So we'll try to clarify any 
confusion. Now in the first paragraph you describe 
the Zeger models as simply attempting to estimate the 
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difference in health care expenditures through 1996 
between smokers, current and former, and never 
smokers of similar background characteristics, such 
as age and sex, regardless of when or why that 
smoking occurred. Now take that as your task, that 
you want to create a model that answers that 
question. 

A. Okay. 

Q. Okay. 

A. Okay. 

Q. All right. 

A. So just to be clear, I'm — I'm supposed to 
build a model that addresses this descriptive 
question of just comparing average health care costs 
between smokers and nonsmokers with similar 
background characteristics, such as age and sex, 
without any causal implications at all? 

Q. All right. 

A. Okay. 

Q. All right. I'm asking you to construct a model 
based on your description of what the Zeger models 
do. 

A. Okay. 

Q. Okay. Now let me ask you this: Would your 
first step to be — be to put together a — a team of 
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experts in various fields? 

A. Yes, although this question is certainly a 
simpler question than the causal question so I 
wouldn't — wouldn't be quite as concerned about the 
variety of people on the team. 

Q. Given that it's a simpler question, what kinds 
of experts would you want on your team? 

A. Well, the question is — is purely a descriptive 
one. So the definition of what background 
characteristics are in there are sort of arbitrary. 

I mean it's defined by, well, you want to look at 
people who are similar. What do you mean by 
similar? Well somebody's gets to define — it's just 
a descriptive question. They get to decide what they 
— what they want to look at. It has nothing causal 
in it at all so they — they can define whatever 
background characteristics they want. 

Q. Would you contact any experts to model that 
question? 

A. Well, somebody has to tell me what the question 
is . 

Q. Well I've just described to you — 

A. Let me — 

Q. — what that question is. 

A. No, you haven't, and I was going to say why. 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

294 

You have to tell me which background characteristics 
you want me to control for. Because there's no 
selection of those background characteristics that's 
driven by any causal concern. You've told me that 
there is no causal concern, so therefore it's purely 
a description and you have to tell me what — what 
you want me to describe. I don't get a team of 
experts to find out what you want me to describe. 

You get to tell me what you want me to describe. 

Q. And are you saying that this description of the 
Zeger models is — is not sufficient for you to do 
any modeling? 

A. Well if you told me what, such as age and sex, 
if you went on and told me you want age. Do you just 
want age and sex or do you want age, sex, and alcohol 
consumption? Or age, sex, and seatbelt use? Or age, 
sex and race? Or age, sex, and income? Or age, 
sex? Tell me — tell me what the list is, just 
specify the list, and then I can go ahead and do it. 
Q. Would you make an independent determination 
about the variables or would you look to an expert? 

A. There is no expert involved in this. It's the 
person framing the question. It's just a do you want 
to have average incomes by age and sex? What's your 
question. Or do you want to have average income by 
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— by only by race? It's just a descriptive 
question so you don't ask an expert what the question 
is. You get to — 

Q. So — 

A. You get to — you're telling me you want me to 
answer a question, a purely descriptive question, 
then you — you have to tell me what question it is. 
Q. So as you describe — 

A. I would go to an expert to — 

Q. — the Zeger models here, you could not do a 
model? 

A. Yes, I could. You have to tell me what the 
question is. If you tell me it's age — can we just 
say age and sex? You want me to compare health care 
expenditures, the difference in health care 
expenditures between smokers and never smokers of the 
same age and sex. Is that the question? Sure, I can 
do that. The same age and sex and income level, yeah 
I can — I can build a model to do that. You say 
age, sex and education level and income, yeah. I can 
do that. You just have to tell me which of those you 
want. You don't tell me which of those you want, how 
could I do it? That's like — like saying "Please 
answer the following question;" but you have to frame 
the question. I — am I — you know, I — if you 
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1 frame the question with what the background variables 

2 are, then sure, I can go ahead and build a model to 

3 — to try to address that. 

4 MR. BIERSTEKER: You want to take a short 

5 break? Just for the — it's for the necessary. 

6 MR. HAMLIN: Sure. Okay. 

7 MR. BIERSTEKER: Thanks. 

8 (Recess from 10:02 to 10:06 a.m.) 

9 BY MR. HAMLIN: 

10 Q. Professor Rubin, if you could turn to page 6 of 

11 your report. 

12 A. Okay. 

13 Q. Under Roman numeral III you state that "The 

14 Zeger Report appears to be an attempt to address a 

15 different causal question," and you described the 

16 causal question as follows: "What health care 

17 expenditures were incurred by plaintiffs as a result 

18 of the existence of cigarette smoking, regardless of 

19 defendants' alleged wrongdoing." 


20 


Could 

you model that question? 

21 

A. 

Yes. 

It would be more difficult. I suppose 

22 

Q. 

What 

would — what would your first step be? 

23 

A. 

Well, 

you'd have to try to find data that 


24 addresses behavioral changes for people who are — in 

25 the current world who are smokers, what they would 
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have done instead of smoking, what — what their 
behaviors would be like. 

Q. All right. And as you sit here today, as you've 
already testified, you — you can't identify a 
specific data set; right? 

A. No, I cannot. 

Q. What would your next step be? 

A. Well, the — if I had data to — of that — of 
that kind, I would — it would have features to it 
that there'd be probabilities for them doing other 
kinds of things instead of — instead of smoking. 
Perhaps they would be involved in other risky 
behaviors or perhaps they would over-eat more or 
perhaps they would be attracted to illicit drugs or 
perhaps they would be more attracted to behaviors 
such as gambling that have risks associated with them 
or perhaps they'd be more likely to engage in other 
kinds of risky behaviors, I don't know, but it would 
have to — that — that data would then be used to 
model what those other behaviors would — would be 
like. And it'd have a probability distribution over 
the course because you wouldn't know precisely what 
they would be like. 

Q. Would you attempt to contact an epidemiologist? 
A. Sure. But for this — for this phase of — of 
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behavioral changes there'd be other people who might 
be just as — as relevant, psychologists or 
sociologists or even economists who have some — some 
knowledge of where data might be available to address 
these kind of alternative behaviors that could exist 
if smoking were not available. But certainly an 
epidemiologist would play a role, as well as other 
people would. 

Q. You would want on your team, I take it, a health 
economist who had some experience with health care 
expenditures for tobacco-related disease? 

A. Absolutely. 

Q. And as you indicated, you yourself have some 
expertise in biostatistics; right? 

A. Some, in certain aspects of biostatistics. 

Q. Would you want someone whose career and 
expertise were focused solely on biostatistics as 
part of your team? 

A. Sure. I would like to have the largest group of 
the best and the brightest people I could find in — 
in covering all the — all relevant topics. 

Q. Now assuming that you have this data and you 
have these people in place, would you attempt to 
model this problem by using a regression analysis? 

A. Well certainly in the general use of the word 
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"regression" in statistics, there would be some 
models that are — that would be in there that would 
be regression models. I mean in a general sense what 
a regression model is is modeling the conditional 
distribution of some outcome variable given some 
predictors, and that's — so there are different — 
there's many — I don't know if you meant by that 
least-squares regression or logistic regression, 
probit regression, or generalized, in a general sense 
of the word, yes, you just — if you just say a 
modeling, I'm trying to predict something from 
something else, in that use of regression, that's the 
game. 

Q. Can you — 

A. But these particular models, the regression 

models used by the Zeger et al. report, I have no 
idea. I doubt it. 

Q. Do you have any idea at all as to what models 
you might use? 

A. Well there's — there's just a huge family of — 
of distributions that can be used. I would certainly 
consider starting out with simpler kinds of 
regression models and — and I have no doubt — and 
it depends so much upon what the data would be like. 
Because presumably you'd have to piece together 
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different data sets to — to create a combined data 
set that could be — could be useful, and so it would 
be — no doubt would be relatively complicated, but I 
— Yeah, I mean there's a collection of regression 
models that certainly would be — would be used. 

Could be with logistic regression. I don't know 
whether you'd use any probit regressions or not. 

There could be squared regressions. There'd be 
general logion models, there'd be hierarchical 
models, there'd be probably some Bayesian smoothing 
of things across different data sets on them. 

Q. Now you said that you would probably have to 
piece together different data sets. 

A. Uh-huh. 

Q. Does that problem impact the decision on the 
kinds of regression or other modeling techniques that 
you would use? 

A. Well yes and no. When you have to piece 
together information from different sources, it's 
kind — there's something that's called record 
linkage, there are different data sources that have 
to be put together, and one way to think about that 
is — can be viewed as a massive imputation problem, 
trying to create data sets that have all the 
information, and so it's not uncommon in certain 
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government settings where that has to be done, to 
have to create this bigger data set from other 
sources. And in doing it you have to make 
assumptions about certain kinds of conditional 
independence assumptions typically, and that means 
that you have to maintain through the analysis the 
awareness that that assumption — that those kinds of 
assumptions have been used, and ideally you should 
reflect that in all the analyses. For example, you 
could — you can do it by the use of multiple 
imputation to reflect the uncertain logit model 
assumptions. There's an article somewhere in my CV 
where I actually did that. It was published I think 
in Journal of Business and Economic Statistics. And 
because of that sensitivity and the extra uncertainty 
of putting these databases together, you have to be 
careful about leaning too hard on assumptions and 
further models, you have to be so careful about it, 
so yes and. The answer to your question is would it 
affect the particular models I used? I don't know. 
But it certainly would affect the way I looked at the 
results of the models, because that would be — I'd 
do many — I had do more things, I would be more 
sensitive, I would do more sensitivity analysis, I'd 
be aware of the fact that this information was coming 
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from different databases, and therefore there are 
these implicit assumptions that I want to make 
explicit and see what affect they have on — on 
subsequent analysis. That was a long answer to the 
question, but it's a complicated question, issue. 

Q. I take it that piecing together different data 
sets introduces a certain amount of uncertainty; 
right? 

A. Relative to if you had all the information on 
everybody in one database, that's correct. 

Q. Right. And I think what you — what you said 
was that it wouldn't — or that uncertainty would not 
necessarily influence your decision as to the kinds 
of models that you would choose, rather it would 
influence your review of their results? 

A. That's a fairly accurate description of what I 
said, yes. 

Q. And you — 

A. I wouldn't say it quite that way, but it's 
certainly — it's close enough, I'm not going to 
argue with you about it. 

Q. And you said that you would do more sensitivity 
analysis in a situation where you had to piece 
together different data sets? 

A. Correct. 
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Q. Can you give me some examples of the sorts of 
sensitivity analyses you have in mind? 

A. Yes. In fact if I had that article in front of 
me, I could — I could give you an exact example. In 
— when you have, let's say, two databases and they 
have one set of variables that are in common, let's 
say X, there's a big collection of variables called 
X, and the other database has Y, a collection of 
variables called Y, and the other data base has a 
collection of variables called Z, but — so there's 
no person, there's no unit in either database that 
has both X, Y and Z, so background variables, smoking 
behaviors and expenditures, just roughly, but you 
don't have a database that has both smoking behaviors 
and expenditures, let's say, but you do have X in 
common to — to both databases. Then in order to — 
but you — but the question that you're addressing 
involves some relationship between Y and Z. Or maybe 
X and Y and Z, involves all three variables, but 
let's just suppose for just simplicity it just 
involves the relationship between Y and Z. Then in 
order to address that question with a data set like 
this that has no joint observations in Y and Z, you 
have to make an assumption, and what that assumption 
is and what that assumption concerns is the 
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conditional relationship between Y and Z, given X. A 
common assumption, but not necessarily valid, is that 
given X, Y and Z are independent. And then you can 
go off into simpleminded imputations or something or 
modeling assuming that they're conditionally 
independent given X. In this paper that I'm 
referring to that was the topic, and we — I 
investigated sensitivity to that assumption of 
conditional independence by creating multiple 
imputations under different assumptions about the 
conditional relationship between Y and Z, given X. 

So I first assumed they're independent, conditionally 
given X, which would be a zero partial correlation 
between Y and Z. Then I assumed they were correlated 
.1 and I saw how the answer changed, the answer 
being — I don't remember what the answer was, maybe 
it was the average Y or the average Z or the average 
Z conditionally in cells given Z. I don't remember. 
And then I could create imputations of the whole 
combined data set assuming the partial correlation 
between Y and Z was .5. And then I assumed it was 
point 8. And then you saw how the answers changed. 
And if they changed dramatically across these 
different assumptions about the partial correlation, 
then the answer is you don't know what the answer is 
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— the conclusion is you don't know what the answer 
is unless somewhere you get some information on the 
partial correlation between Y and Z, given X. 

Someone would say. Well, look, I really understand 
this problem, they really are conditionally 
independent, this is an issue I truly understand; 
then that becomes the — then if you have agreement 
on that, that there was a consensus of that very low 
partial correlation, then you have an answer under 
that assumption. So I mean if we — if I — I could 
point that article out to you if you'd like me to. 

Q. Well would you do any other sensitivity 
analysis? 

A. Yes, I would frequent — within — that would be 
the primary one because that's an assumption for 
which I've assumed there is no data to address it, 
but I would do the usual kinds of complete-data 
sensitivity analysis as well. If I'm — if the 
question involved doing a linear regression, if the 

— sorry. Let me start again. 

If the question involved a prediction and I 
first addressed that prediction by doing a linear 
regression, then I would probably do a sensitivity 
analysis to the linearity assumptions, perhaps 
putting in quadratic terms or having a different 
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error distribution, the usual sort of things that you 
should be doing to investigate how well a model — a 
model fits. And depending upon the question that's 
being asked there are different things that can be 
done. 

Q. Can you think of any other sensitivity tests 
that you would want to run in a situation where you 
are piecing together different databases? 

A. Well there are assumptions — I mean this is 
such an open-ended question it's hard — it's hard to 
address, but I'll — I'll try to be helpful. 

Q. Well what I'm — what I'm asking you is. If you 
were charged with this task of modeling this causal 
question on page 6 that we've talked about, and you 
had to piece together data from different databases 
and you created a model, what sensitivity test would 
you run? 

A. Okay. Oh, well, there certainly would be other 
sensitivity analyses regarding the assumptions that 
would have been made in the model regarding other 
behaviors that would take place in the absence of 
smoking. Depending upon how much or what kind of 
data are available to address that, it may be 
necessary to make larger, more substantial 
assumptions, and then there would be certain — have 
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to be sensitivity investigations for those 
assumptions. If somehow the — there really is a 
great data set somewhere or combined data set 
somewhere that addresses the question of how people 
might behave — smokers might behave in the absence 
of smoking, then there wouldn't be as much 
sensitivity to those assumptions. But it just 
depends upon the combination of data and assumptions 
that this team of people could put together. 

Q. Now, with respect to selecting appropriate 
models or statistical methods to address this 
question, do you have in mind a specific model or set 
of models that you would use? 

A. Well I have in mind the whole collection of 
models that I — that I use and know about. I sort 
of described them before in some general kind of way. 
Q. What I'm asking is: Do you have in mind a 
specific idea of what models you would use to address 
a question such as the causal question you pose on 
page 6 which you say Zeger answered — or attempted 
to answer? 

A. Well, no, I didn't — I didn't — okay — what I 
said here "appears to be an attempt to address a 
different causal question". I don't think they even 
say that anywhere, but there's the implication that 
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if it's — if it's — if we're to regard that report 
as at all relevant to the question, then I — this 
sentence just means I guess that's what they thought 
they might have been doing, although they don't — 
they kind of deny that in the report and — and then 
in deposition, so having read depositions now I 
probably wouldn't — wouldn't write it that way. 
Because they — they don't say that, so this was — 

Q. Well what is your understanding of what they 
did? 

A. My understanding of what they did is — it's 
very hard to understand what they did in some ways, 
but it's — it's this descriptive question. They 
define something called Smoking Attributable 
Fraction, SAF, and they attempt to estimate that in 
different subgroups defined by age and sex and 
sometimes by something else, insurance type and 
sometimes other covariates as well they bring in, so 
just it's kind of a descriptive trying to estimate 
what this thing they — SAF, which is just a 
definition like I can define something to be the mean 
or the median, give it a name, they defined this 
quantity, then they try to estimate it in different 
subgroups where the subgroups differ depending upon 
which kind of health care, which kind of diseases 
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they're looking at, so it's — it's just — it's an 
attempt to — to describe something. 

Q. All right. On page 6 in the second full 
paragraph under Roman numeral III you state in part 
that "the Zeger report does not reliably estimate the 
difference between plaintiffs' estimated health care 
costs today and what they would have been in a 
hypothetical world in which nobody ever smoked." 

Do you see that? 

A. In which nobody ever smoked. Oh, at the 
bottom. 

Q. Yes. 

A. Yes, I see that, and I guess I don't necessarily 
agree with that. 

Q. Now if you were going to — well first of all, 
you have not attempted to estimate the difference 
between Plaintiffs' estimated health care costs today 
and what they would have been in a hypothetical world 
in which nobody ever smoked; right? 

A. Neither I have, nor has anybody else. 

Q. But you have not done that in this case; right? 
A. No, I have not. 

Q. And as you say, you don't know of anyone else 
who has? 

A. No. I don't even know of anybody who — who 
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thinks they have. 

Q. Now if you were going to attempt to reliably 
estimate this difference, would you follow the same 
procedures that we just previously talked about; that 
is, to put together a team of experts first? 

A. Yes. 

Q. And then look to these data sets that you've 
described; right? 

A. Look to these kinds of data sets that I have 
described, but rely on people who really knew about 
data sets to — to put them together; correct. 

Q. And then, thirdly, you would consider different 
types of statistical methods to estimate the 
difference; right? 

A. The — right, you first estimate behavioral 
changes in a world without smoking, and then another 
part of the model would be relative expenditures for 
the — for — in this world without smoking for these 
changed — changed behaviors, and which kind of 
models I would — I would use would depend on what 
kind of data my team could find. 

Q. But as you sit here today you — would you agree 
that the model would most likely take the form of 
some form of regression analysis? 

A. As we said before, using the interpretation that 
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regression analysis is just trying to predict 
something from some other things that we're talking 
about predicting, so — 

Q. And then you would attempt to perform these 
sensitivity analyses; right? 

A. Correct. 

Q. And would you also attempt to compute confidence 
intervals? 

A. Confidence intervals would — are part of the 
estimation under the model; in other words, you — 
under — under a particular model, regression model 
or any other kind of model, you have an estimate and 
an interval that — that goes along with that 
estimate in which you think the thing lies, and so 
the sensitivity analysis would be to how those 
intervals move around, so the — the point estimate 
and the interval around it moves from under — as you 
change assumptions. So certainly intervals would be 
— confidence intervals would be a part of the — of 
the result of the — of the exercise. 

Q. And confidence intervals basically tell you 
what? 

A. Are you asking for a formal definition? 

Q. (Nods head.) 

A. Okay. Let's suppose it's a 95 percent 
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confidence interval which is a common — 
commonly-used one. They address the following 
question: Confidence interval under a — under a 

model — let's talk about — there's a true value of 
the thing you're trying to estimate, it's unknown, 
but it's out there somewhere is the — is the story 
that you start with, and you have a — have a data 
set that addresses the estimation of that quantity, 
that unknown quantity, and you now — from that one 
data set you now estimate that quantity and you also 
have a confidence interval about that quantity, so 
it's basically the estimate plus something and minus 
something, so it's a range of values for that unknown 
quantity, and if it's a 95 percent interval, what 
that says is that if I now went back in hypothetical 
worlds over and over again collecting data like this 
data but not the same data, but by the same procedure 
as I collected the data set in front of me, but 
hypothetical new data sets, over and over again, and 
each time I had a hypothetical data set I did the 
same calculation I did in this data set, used the 
same method to get a point estimate and the 
confidence interval, the point estimates would change 
and the confidence intervals would change across all 
these hypothetical data sets, that in the — but the 
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— the interval that I have in my data set is a 95 
percent interval if in at least 95 percent of those 
hypothetical data sets the confidence interval that's 
created in those hypothetical data sets, if at least 
95 percent of them include the true unknown value. 
That's the definition that Neyman made I think in 
1934 where — when he invented the term. It's often 
misstated to be exactly 95 percent of the data sets, 
but that's not the way he defined it. He defined it, 
so he gets to — he gets to keep the name. 

Q. Now as you sit — 

A. So it's really the idea of hypothetical 
replications of the same process of the way you get 

— got the data, what would happen in long-run 
hypothetical replications. 

Q. Now as you sit here today you have no results of 
any model to compare to the results of Zeger, et al.; 
right? 

A. You mean — I do not have any results of 
analyzing the — that's not right. I'm not sure what 
the question means I guess. 

Q. Okay. You yourself haven't done a model to 
determine health care expenditures for smokers; 
right? 

A. Correct. 
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Q. And so you have no results regarding whether 
there may be an increase or a decrease in health care 
expenditures because people smoke; right? 

A. Correct. 

Q. And so you have no results to compare or 
contrast with those of Zeger, et al.; right? 

A. I have not done any analyses — 

Q. Right. 

A. — on the data to come up with dollars, 
amounts. I have not done that. 

Q. And so you have no dollar amounts to compare to 
those in the report by Zeger, et al.; right? 

A. I have not done any — any analyses of their 
data, and their — with dollar amounts in those 
analyses, so I have no analyses of dollar amounts 
from that data set to compare with their dollar 
amounts from that — from those data sets. 

Q. Now have you talked to Dr. Foster who is another 
expert for the Defendants in this case? 

A. No, I don't even remember seeing his name. 

Q. So you have not — 

A. I probably have — have seen his name, but I 
certainly have never talked to him. 

Q. So you have not reviewed his report regarding 
the Plaintiffs' damage model in this case? 
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A. No, I don't think I have, unless I'm becoming 
senile. 

Q. Okay. Let — let's assume that Dr. Foster 
attempted to create a model of basically health care 
costs attributable to smoking. All right? 

A. So you want me to assume he built a model trying 
to estimate this thing called SAF within the same 
kinds of sells defined by covariates as — no, that's 
not right. 

Q. No. Just — just assume that he made an effort 
to create a model to predict or estimate health care 
costs for smokers. Just — just assume that as an 
initial proposition. All right. 

A. I'm not sure what that means, but I'll try to go 
with it and see where — see what happens. 

Q. Also assume that he did not use — Strike that. 

Assume that he used some data sets with missing 
values. 

A. Okay. 

Q. All right. And further assume that he did not 
use multiple imputation to address those missing 
values. 

A. Okay. 

Q. And further assume that he did not use 
propensity scoring. 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

316 

A. Okay. 

Q. Would you conclude that his results were 
unreliable because of his failure to use multiple 
imputation and propensity scoring? 

A. Well, I don't even know what he's trying to 
estimate, but I — I'll try to answer the question. 

As we discussed earlier, there are other valid ways 
to handle missing data other than multiple 
imputation, so the fact that he did not use multiple 
imputation does not necessarily mean that he made a 
mistake. 

Q. Is that also true for propensity scoring? 

A. That's also true for propensity scoring; just 
because he didn't do it, that by itself does not 
necessarily mean he made a mistake. But I'm 
certainly not — since I don't know what was done, I 
— I have no opinion on whether it was reliable or 
not because I don't even know what question is being 
addressed by the — by the analysis or the model. 

Q. Now you've talked about estimating health care 
costs by comparing a real world to a — to a 
hypothetical world. Correct? 

A. Correct. 

Q. Now could you also estimate health care costs 
for smoking by using a real world real population and 
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comparing real smokers to real nonsmokers controlling 
for variables like age, sex, education and other 
demographics? 

A. Well that's a description of the kinds of 
analyses you might want to do to try to address the 
question of what would happen in this other world in 
which either there were no smoking or the Defendants 
didn't commit the alleged fraud. You analyze data 
that comes from the real world, but the question 
you're addressing is a question from a hypothetical 
world, in the same way as a confidence interval 
addresses questions across many hypotheticals, 
samples that you took. You take the one sample and 
you do something with it, and then you — you think 
about all these hypothetical other samples that you 
might have had. That's what statistics is about. 

You — 

Q. So you're saying that it is — 

A. You can't — 

Q. — possible then to do an estimate of health 
care expenditures for smoking by comparing real 
smokers to real nonsmokers? 

A. No, I'm saying that's a part of the — of the 
analysis, that information is part of the information 
that you — that you would have to use. Correct. 
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Q. And you — 

A. You have to use real data and real data on the 
health care costs of smokers versus nonsmokers does 
come from real data on smokers and nonsmokers, but it 
doesn't say how you use that information. 

Q. Are you — once you have the data on the real 
smokers and real nonsmokers, then you would control 
for various variables such as age, sex, education and 
other demographics in order to come up with your 
result; right? 

A. Yes and no. It depends when those variables are 
recorded and how it — how it's done. It's not just 
the simple matter of — of saying I would control for 
these things, but it's — it's how you'd control for 
them, and which — which versions of these variables 
you're trying to control for. 

Q. Do you know of any researchers that have 
followed that procedure in determining health care 
costs for smoking? 

A. Health care costs for smoking. Not in the way 
that I'm thinking about it should be done. Certainly 
in — in other observational studies people have done 
things like that. 

Q. Can you give me an example? 

A. Of ones I've been involved in? 
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Q. Any observational study that would meet the 
criteria that we just discussed. 

A. Well the one I talked about yesterday involving 
breast cancer, that's an observational study where we 
controlled for — for background characteristics. 
There's a study I've been on and off involved with 
for years, not recently, but a while ago, on trying 
to estimate the effects of intrauterine exposure to 
hormones and barbiturates on kids' psychological life 
later, intelligence and behavior as they — as 
teenagers and as young adults. Observational studies 
on the effect of exposures to drugs. Estimating the 
effects of exposures in randomized experiments where 
there are noncompliance. You have to worry about 
these kinds of issues, such as the effect of — well, 
I won't go on, but there are lots of — there are 
just examples all over the place where you have to 
adjust. 

Q. So it's a commonly-used technique? 

A. To adjust for proper covariates in the analysis 
of observational data and attempt to estimate the 
causal effect of something, yes, that's what you — 
that's one of the things that you have to do. 

Q. By "adjust for these covariates" do you mean 
controlling for those covariates? 
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A. Yes. 

Q. And by "covariates," that would include 
variables like age and sex and education and — 

A. Yes and no. 

Q. — other demographics; right? 

A. Yes and no. What defines a proper covariate is 
something that — whose — whose value you — took 
place — whose value is obtained before the treatment 
you're trying to understand the effect of. So in 
this context of the causal effect of alleged 
misconduct in 1965, let's say, then the covariates 
would be things that were measured before 1965, or 
variables that were measured after 1965 but could not 
have been affected by the alleged misconduct or not, 
so their value is the same as it would have been 
before. So those are proper covariates. 

Q. But age and sex could be considered as a 
covariate or covariates in a study? 

A. Well year of birth is better than age because 
people die, so that year of birth takes its value 
before. I mean age not necessarily because age can 
be dead, you know, one of the values of age is dead, 
and that can be affected by alleged misconduct, 
whereas year of birth cannot be, and sex cannot be, 
race cannot be, education up to 1965 cannot be, 
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income up to 1965 cannot be, parents' health up to 
1965, your smoking habits up to 1965 cannot be 
affected, your drinking habits up to 1965 cannot be, 
you're psychological state, psychiatric expenses, 
medical expenses up to 1965, all of those are proper 
covariates. Things that happened after 1965 are 
outcomes. 

Q. I'm not asking about any specific model. All 
I'm asking you is whether a variable such as age can 
be considered in a — in a linear regression as a 
covariate? 

A. I just told you, that it's not as good as year 
of birth. Because it can — if there's an account 
made for death, then it is — then it's a — can be 
considered a proper covariate, because then it goes 
back to year of birth, it converts to year of birth, 
but then you have to have people who know and were 
alive in the database as well because otherwise it's 
an outcome. 

Q. But sex can also be considered as a covariate; 
right? 

A. Yes, for people — for people who are born 
before 1965, yes. 

Q. I'm not talking about any specific model, I'm 
just talking about in a — in a general sense these 
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1 items like age — 

2 A. Right. 

3 Q. — and sex and education and — and demo — 

4 other demographics are considered to be, quote, 

5 "covariates" or variables; right? 

6 A. And I'm saying no to that and I'm telling you 

7 why I'm saying no in general, because they may or may 

8 not be depending upon what — what analysis we're 

9 going to do for what kind of causal effect. 

10 Q. Okay. 

11 A. Often they are. What you're saying is 

12 absolutely correct, and often they are, but it has to 

13 be done carefully. 

14 Q. Let's turn to page 7 of your report. 

15 A. Okay. 

16 Q. You give another example of a world with smoking 

17 and a world without smoking. Do you see that? 

18 A. Yes. 

19 Q. And In your world with smoking this smoker was 

20 born in 1935, started smoking heavily In 1955, and In 

21 1980 he contracts lung cancer and dies. 

22 Do you see that? 

23 A. Yes. 

24 Q. In the world without smoking this person does 

25 not smoke but over-eats, and in 1975 becomes severely 
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overweight, with high cholesterol, and in 1981 has a 
heart attack, 1985 has bypass surgery, and then he 
has a second heart attack, and in 1995 he's still in 
the nursing home. 

Do you see that? 

A. Yes. 

Q. Now let's go to the world with smoking where 
this person contracts lung cancer and dies. Now what 
is — Strike that. 

Is the basis for your hypothetical essentially 
the Surgeon General's report and conclusion that 
smoking causes lung cancer? 

A. Well since I've never read that report, it's — 
the basis would be more general just understanding of 
what the issues are in this case regarding smoking, 
and that certainly seems to be a potential likely 
outcome relative to other kinds of outcomes, that I 
just chose it because it's — I — as yesterday you 
asked me whether I — I believe in a nonexpert way 
that — that the primary cause of lung cancer is 
smoking, and I said yeah, I believe that. I don't 
have an expert opinion about it, but it's based on 
that. 

Q. Yeah, I mean this is based on your belief that 
cigarette smoking causes lung cancer; right? 
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A. It's a belief, yeah. 

Q. All right,? 

A. To try to make an example that looks something 
plausible — somewhat plausible I mean. 

Q. All right. Now let's look at what Zeger, et al. 
do in a world with smoking. They count up the 
smoking-attributable health care costs in a world 
with smoking and no dollars are counted after 1980; 
right? 

A. I'm not sure I understand the question. 

Q. Okay. Let — let me just simplify it. 

A. Okay. 

Q. In your opinion does — or do Zeger, et al. 
count any dollars after 1980 in this world with 
smoking that you have hypothesized? 

MR. BIERSTEKER: Object to form. 

A. Count any dollars. Well as — there are no 
dollars spent on this person after 1980. So in the 
databases they use there would be no costs after 
1980, so therefore they do not use costs after 1980 
for this person, if this person were in that 
database. 

Q. Okay. Now Zeger, et al. would not claim any 
health care costs in this world without smoking that 
you have created; right? 
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A. I don't know what they would do. 

Q. Well just based on your understanding of the 
model, if there's no smoking, then there would be no 
health care costs for smoking-attributable disease; 
right? 

MR. BIERSTEKER: Object to form. 

A. They don't attempt to address this question. So 
you're saying — you're asking me something that's 
contradictory, saying — are you asking me to assume 
they address this question and they don't, they say 
they don't, and then you're asking me what they do 
when they don't address this question. I'm not sure 
of what you're — 

Q. Let's simplify it. 

A. Okay. 

Q. As you say, they don't address this question; 
right? 

A. Correct. 

Q. That's all I was attempting to get out. 

A. Fine, they don't address the causal question, 
that's correct. 

Q. They don't attempt to address this question of 
what would happen in a world without smoking? 

A. That's correct, they just try to estimate this 
thing called SAF in various sells. 
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Q. And you state that "the Zeger Report would give 
no consideration at all to the differences in health 
care costs borne by plaintiffs after 1980." 

Do you see that at the bottom of the — 

A. Yes. 

Q. — page? All right. And that's because this 
person died in 1980; right? 

A. Well that's one of the reasons in this example, 
that's right, but they don't even attempt to do that 
so they — they don't — 

Q. Well you agree that the Zeger report would give 
no consideration at all to the differences in health 
care costs borne by Plaintiffs after 1980 for the 
person in the above — 

A. Correct. 

Q. — example; right? 

A. Correct. 

Q. And you — and you also agree that at least as 
to in the world with smoking in that example there 
were no costs after 1980; right? 

A. Correct. 

Q. Then you say on page 7 that the passage of time 
must be taken into account. Do you see that? That's 
under the example. 

A. Yeah. Yes, I do. 
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Q. Now in your example the smoker died in 1980 
because he smoked, according to your hypothetical; 
right? 

A. Because in the sense that in this comparison he 
would not have died of lung cancer in a world where 
— without smoking. 

Q. All right. Then in your world without smoking 
the individual lives beyond 1980 and he incurred 
further health care costs; right? 

A. Right. 

Q. And in fact I think page 8 you further 
hypothesize that he went into a nursing home paid for 
Medicaid and spent tens of thousands of dollars; 
right? 

A. Yes. 

Q. Let's take that entire hypothetical where we 
compare the world with smoking and the world without 
smoking. 

A. Okay. 

Q. Is it your testimony that the tobacco companies, 
the Defendants in this case, are entitled to a credit 
because they killed off this person in 1980 and saved 
the state money? 

A. I'm — I'm saying that if the question is how 
much — for this hypothetical person how much money 
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was spent on health care costs, this is the way you 
have to do the calculation. That's what I'm saying. 
Q. Well are you saying that the tobacco companies 
are entitled to a credit because their product kills 
people early and thus saves the state money when 
people are no longer around and therefore do not go 
into the nursing homes? 

A. I'm — 

MR. BIERSTEKER: Objection; asked and 

answered. 

A. I'm saying that if you're going to ask the 
question how much money and how much was spent in 
health care costs for this person, accepting this 
hypothetical as being true, with and without smoking, 
you have to answer that question, and the answer to 
that question is a comparison of health care costs in 
these two scenarios. 

Q. Well would you agree that the end result is that 
in the world without smoking there are more costs 
involved; right? 

A. In the way I set up the hypothetical? 

Q. Yes. 

A. In the world without smoking in this example, 
the way the hypothetical is set up, there are more 
costs. 
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Q. Right. 

A. Correct. 

Q. And would you then conclude from that that the 
tobacco companies then ought to be credited for 
saving the state money in a world where there is 
smoking? 

MR. BIERSTEKER: Object to form. And asked 
and answered. 

A. I'm saying that if the question is what are the 
differences in costs, then the answer is these are 
the differences in costs. That's just the answer. 

Now if you don't like the answer, then — then you 
should ask a different question. I mean if they — 
if the question being — being posed by the 
Plaintiffs is what is the difference in costs, and 
we've both agreed this is the answer, that's the 
answer to that question. What more can I say? 

That's the answer to the question. 

Q. Well you can answer my question, which is: Do 
you believe that the tobacco companies are entitled 
the a credit because their product kills people early 
in life? 

MR. BIERSTEKER: Objection; asked and 

answered. 

Q. I'm just asking you that simple question, 
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whether you believe that or not. 

A. I'm — I'm a statistician. I'm telling you 
we've agreed on what the answer is for this question 
of health care costs. Now if you're asking me what 
other lawsuits should be brought against the tobacco 
industry for alleged misconduct, asking different 
questions having to do with early death, I'm not an 
expert in that at all, I'm not a lawyer, so I don't 
— that's something that's — that I don't even know 
how I would frame that claim. But if the question is 
comparing costs, we've just agreed in the 
hypothetical that there's a difference in costs such 
that the costs are greater in this hypothetical 
example in a world without smoking. If that's the 
answer to that question, then that's the answer to 
the question. I can't — I can't lie and say that 
that's not the answer to that question. 

Q. Well — 

A. Or say I wish it weren't the answer. We've 
already accepted the hypothetical. 

Q. This is your hypothetical, and isn't the — the 
natural conclusion from this hypothetical that the 
tobacco companies are entitled to some kind of a 
credit because their product kills people earlier 
than those people who do not use the product? 
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MR. BIERSTEKER: Objection; asked and 
answered as well as objection to form. 

A. It's not my natural conclusion. My natural 
conclusion is that the answer is what the answer is. 
The question concerns comparing health care costs. 
That's what the hypothetical dealt with, comparing 
health care costs. That's my hypothetical, comparing 
health care costs, and you said accept the 
hypothetical, you told me to accept it in its 
full-blown version, it continues on page 8, and I 
accept it, and the conclusion is that there's a 
difference between health care costs in the worlds 
with and without smoking such that the health care 
costs in the world without smoking in this 
hypothetical example, which you told me to accept, 
are greater in a world without smoking. 

Q. Well this is your hypothetical, I didn't ask you 
to accept anything but — 

A. I believe you did. 

MR. BIERSTEKER: I believe it too. 

Q. — be that as it may, in your hypothetical, the 
reason that there are less costs in the world with 
smoking is because this smoker died in 1980 because 
of lung cancer; right? 

A. That's correct. 
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1 Q. Okay. Now you say that one must also consider 

2 the possibility that smokers' other health-related 

3 behaviors might be different in a world without 

4 smoking, and you give an example of over-eating or 

5 alcohol consumption. Now — do you see that? 

6 A. Yes. 

7 Q. Again — 

8 A. Wait a minute. 

9 Q. Yeah. 

10 A. Let me see if I can find the exact — the exact 

11 place. 

12 MR. BIERSTEKER: It's page 7. 


13 

Q. 

Actually it's 

page 7. 

14 

A. 

Right, 

yes, I 

'm with you. 

15 

Q. 

Okay. 

As you 

sit here today, can you identify a 


16 database that would provide this information about 

17 smokers' other health-related behaviors in a world 

18 without smoking? 

19 A. I cannot at the moment identify an off-the-shelf 

20 database that addresses that, but I do believe that 

21 — that data sets exist, and again I would — I 

22 would go to people who know more about that than I do 

23 as a starting place. 

24 Q. Now, let me direct your attention to the bottom 

25 of page 8. You state that if you were to design a 


STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

333 

statistical analysis from observational data to 
address the question of what health care costs over a 
number of years are caused by a world where smoking 
exists, you say you would attempt to include a model 
of health-related behavior when smoking is not 
available. Now have you asked the tobacco companies 
if they have such data? 

A. No, I have not. 

Q. Have you asked the tobacco companies if they 
ever ran such a study? 

A. No, I have not. 

Q. Now you go on to say that perhaps data on 

behavioral changes when former smokers quit smoking 
or data on behavioral differences in environments 
where cigarettes are more or less available could be 
used for that purpose. 

Can you identify for me an environment where 
cigarettes are more or less available? 

A. Well, one thought would be perhaps places where 
taxes are higher or changes in time when taxes get 
higher that the cigarettes are sort of less available 
because they're more costly in that sense; maybe that 
could be used to address it. Maybe there are 
countries or regions where cigarettes are much less 
available than they are generally in the United 
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States. Maybe there was a time in some countries 
when cigarette smoking was prohibited by law, there 
was a prohibition against it. That's the kind of 
think I had in mind, so again I'd have to ask 
somebody who knows something about databases like 
that — like that. 

Q. All right. Now you say that you would look for 
data on behavioral changes when former smokers quit 
smoking. 

A. For example. 

Q. As an example. And you would use this to create 
this hypothetical world where there was no smoking; 
right? 

A. I would use it to help model some aspects of 
that hypothetical world without smoking. 

Q. Now this — it's clear that this data on 
behavioral changes when former smokers quit smoking 
has in it the fact that people smoked; right? 

A. Correct. 

Q. I mean you don't know of a database that has 
data from a world where no one smoked; right? 

A. As I said, I do not know of any, but it's — 
it's possible that there do exist environments 
perhaps where cigarettes were illegal or very hard to 
come by that would — it could be used to address 
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that — that question, but I do not know of — of a 
data set here, as we sit here today, that I can grab 
off the shelf to address that. I do not. 

Q. Now, you — in your next paragraph on page 9 you 
characterize the Zeger report. Do you see that? 

A. Yes. 

Q. And you say that Zeger limits the analysis to 
only the actual recipients of health care by 
Plaintiffs in a world with smoking. Do you see that? 
A. Yes, I do. 

Q. And — and that's your testimony today; right? 

A. Correct. 

Q. And you say that Zeger ignores possible changes 
in the populations of the recipients of health care 
funded by Plaintiffs; right? 

A. Correct. 

Q. And by that you mean changes that might occur if 
the tobacco companies' behavior had been different; 
right? 

A. Correct. This was referring back to this 
example where this person in the world with smoking 
may never have — have received medical costs 
supported by the state, whereas in the example he 
does accrue medical costs that were supported by the 
state. So that's just referring — we both agree 
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that that's not considered in the Zeger, et al. 
analysis. 

Q. You also say that Zeger considers only a limited 
number of background variables. Do you see that? 

A. Yes, I do. 

Q. Now you would agree that every study considers a 
limited number of background variables; right? 

A. Limited would be — would be relative, that's 
correct. 

Q. Okay. 

A. But they're all limited in some way. 

Q. They are all limited; right? 

A. Right. I was suggesting that they're limited in 
a way if — if in order to address the causal 
question they — they were — they were limited to a 
set that's not really — really appropriate for 
addressing the causal — causal question. 

Q. Well let me ask you this: What estimates — 
what background variables did Zeger omit that you 
think ought to have been included? 

A. The "ought" makes it sound as if — if he 
included them he — he would get a better answer for 
the causal question, and there's nothing he could 
have done, given the analysis that was — that was 
done to address the causal question. 
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Q. Okay. What variables did Zeger omit? 

A. I don't understand the question. 

Q. You say that he — 

A. There's an infinity of variables that exist, and 
he didn't use an infinite number, he used a few. 

Q. You say he considers only a limited number of 
background variables; correct? 

A. Correct. 

Q. Are there variables that you have in mind that 
Zeger ignored? 

A. If — if the Zeger report — Strike that. 

In order to address the causal question of a 
world without smoking or the causal question 
involving alleged misconduct, the right collection of 
background variables to use are ones that are in the 
world — let's talk about the world in which there's 
alleged misconduct. The right collection of 
covariates are variables that took their values 
before the alleged misconduct, and that would include 
not only age, sex, smoking habits, alcohol 
consumption, medical conditions, education, income, 
all — all before the alleged misconduct. Family 
history of medical conditions, psychiatric 
conditions, all again before the alleged mis — 
misconduct. So the — the correct collection and the 
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formulation, if you're going to address the question 
of the alleged misconduct, you have to formulate the 
problem that way and then — and then try to do the 
modeling as we've been discussing earlier. And 
that's not what — what he — what that report did. 
What that report did, as we — I think we all agree, 
is addressing just the descriptive question. I 
define this thing called SAF and now I try to 
estimate it in various cells defined by certain 
characteristics of people such as age, six, whether 
they use seatbelts or not and stuff like that. It 
doesn't — 

Q. So — 

A. It doesn't address the question. 

Q. So what you're saying is that there should — 
there is this right collection of background 
variables for modeling the causal question that you 
posed; right? 

A. I didn't pose the question, no. I think the 
Plaintiffs posed the question. 

Q. All right. But you believe the Plaintiffs posed 
the question; right? 

A. I think it's written. I think it's in — in 
their complaint. 

Q. Well I'm not going to argue with you about — 
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A. Okay. 

Q. — whether it is or it isn't, but you posed the 
question — 

A. The question when I read — when I read those 
parts of the complaint — 

Q. Right. 

A. — that I — that I see. 

Q. So you posed a particular causal question and 
you believe that that causal question is grounded in 
the Plaintiffs' complaint; right? 

A. They — 

MR. BIERSTEKER: Object to the form. 

A. — posed the causal question. Plaintiffs posed 
the causal question. I think I can find a sentence 
where they pose that question, and I — I read that 
and I accept what the English sentence says, and 
therefore I take that as the question. 

Q. And you believe that given that question there 
is a right collection of background variables; right? 
A. There's not a single right collection, but 
there's a right way to think about what those 
background variables should be. 

Q. And your right way of thinking about them would 
be to identify variables like age, sex, alcohol 
consumption and family history before any misconduct 
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by the Defendants; right? 

A. Correct. That's the right way to conceptualize 
it. Actually how that conceptualization gets 
implemented when analyzing data is another issue. 

But the right way to think about it, to say what 
would I really like to have as far as data, is that's 
the right way to think about it. And then of course 
you have to deal with the limitations of the real 
world data sets that we have available to address 
that, because you can't address the causal question 
without a proper form — formalization of what that 
question is. 

Q. And I take it from your — your prior testimony 
that you have made no investigation to determine 
whether there is a data set that includes these 
variables before any misconduct; right? 

A. No, I have not. 

Q. And you have not done any conceptualization of 
this collection of background variables that you 
claim is the right way to think about it; correct? 

A. Well I think I have. That's what we've been 
talking about and you're talking — you're asking 
questions about what would I do, what kind of data 
sets would I try to get, how would I go after it, 
what kind of models would I use? I think that I 
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would call that thinking about the conceptualization 
of it, so I guess my answer to your question is no, 
because I have thought about it, not as deeply as I 
would have to if I were to really do it. 

Q. That's my question. But you haven't really done 
it; right? 

A. No, I have not analyzed any data, I have not 
made an attempt to get — get data. I have spent 
time thinking about, I continue to spend time 
thinking about how to properly conceptualize it. 
Because it is a difficult problem that probably 
deserves to be addressed properly. 

Q. So just so we're clear, in this sentence where 
you say that Zeger considers only a limited number of 
background variables, you are not claiming that Zeger 
omitted a variable like alcohol consumption and 
therefore the results are wrong? 

A. I am not claiming that, because the implication 
would be if he had included alcohol the analysis 
would then be correct and address the causal 
question. It's beyond that. The analysis cannot be, 
quote, "fixed up" to address the causal question by 
adding or deleting variables that are in there, and I 
think from having read the depositions that they 
agree with that. 
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Q. Now you have not run the Zeger model with any 
additional variables not included by Zeger; correct? 
A. I have not run the Zeger model meaning I have 
not estimated these SAFs that they did; correct, I 
have not. 

Q. Can you identify any study in the field of 
epidemiology where the effect of any particular 
variable has diminished or eliminated the effect of 
smoking on disease or health care costs? 

A. Of any kind of disease? 

MR. BIERSTEKER: I — let me interpose — 
let me — could I have the question back? I'm sorry. 
MR. HAMLIN: Sure. 

MR. BIERSTEKER: I'd like to get it right. 
(The record was read by the reporter.) 

MR. BIERSTEKER: Object to the form, but go 

ahead. 

A. I'm not sure I could identify a study going 
either way on that so — so no, I cannot identify a 
study doing that nor the other way. 

Q. Now on page 9 you state in the last paragraph 
that you would begin your analysis by using 
propensity score methods. Do you see that? 

A. Well let me be sure. 

Q. Yeah. 
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A. Where are we, bottom of page 9? 

Q. Bottom of page 9. 

Specifically you say "I would begin by using 
propensity score methods to assess the degree of 
overlap in the distribution of these background 
variables between smokers and nonsmokers." 

A. Well I see that, but to be clear, you can't — 
you can't omit the stuff that's above it, because if 
I — if I were instructed to conduct the analysis of 
the effect of smoking on Plaintiffs' health care 
expenditures accepting all these limitations, which 
in my view would be improper, so you want me to 
accept that. Is that what I'm supposed to do? So 
basically what I'm saying there this analysis doesn't 
mean — doesn't have anything to do with the causal 
question and it's just a descriptive analysis where I 
want to compare in — health care costs for smokers 
and nonsmokers with — in cells defined by some 
background variables that somebody's got to define. 

Is that — that's what I want? That's what we're 
going to do here? I'm going to say if I were to do 
that, to just address this descriptive question, and 
I'm doing the adjustment for these cells that have 
different numbers of smokers and nonsmokers in them, 
if I were doing that adjustment by linear regression, 
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1 I would not begin by doing it the way they did it, 

2 just to address the — just to address the 

3 descriptive question, but I would begin by using 

4 propensity scores to assess the degree of overlap in 

5 distribution of these background variables between 

6 smokers and nonsmokers. That's the way I would begin 

7 if I were asked to address that descriptive question. 

8 Q. So you say you would not employ the regression 

9 analysis — 

10 (Reporter interruption.) 

11 Q. In your report you say that you would not employ 

12 regression analysis in the manner used in the Zeger 


13 

report to 

adjust 

for 

background variables; right? 

14 

A. 

Yes . 




15 

Q. 

All 

right. 

But 

you go on to say that before 


16 relying on Zeger report's regression you would first 

17 make some assumptions underlying those regressions? 

18 MR. BIERSTEKER: I'll object to form. 

19 A. I don't think you stated that right. 

20 Q. Okay. 

21 A. Let's see. I would begin by using propensity 

22 score methods to assess the degree of overlap in 

23 distributions of these. 

24 Q. You say in your report that you would begin by 

25 using propensity score methods to assess the degree 
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of overlap — 

A. Right. 

Q. — in these background variabies between smokers 
an nonsmokers; correct? 

A. Right. 

Q. Now you haven't done that; right? 

A. I have done a little bit of that, yes. I 
believe that was forwarded to you. 

THE WITNESS: It was, right, Peter? 

MR. BIERSTEKER: It was. 

BY MR. HAMLIN: 

Q. Now — and we'll get into that in a minute. 

A. Okay. 

Q. Would you agree that propensity scores can be 
used to adjust for confounding variables in 
observational studies? 

A. Yes, that's the — in fact that was done in the 
breast cancer example. 

Q. And would you also agree that propensity scores 
indicate a potential bias in estimated treatment 
effects? 

A. I know what you're getting at, but that sentence 
doesn't make sense. I can try to state exactly, 
because I think I know what you're getting at and I'm 
not trying to — trying to duck the question. If you 
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want I can reformulate the question because I think I 
do know what — what you're getting at. 

MR. BIERSTEKER: I guess I better object to 

form. 

A. Maybe it was a well-formulated question. Why 
don't you read it back. I don't think so though. 

I'm certainly not trying to duck it, I will answer a 
version of it, I just want to answer a version that 
makes sense. 

(Reporter interruption.) 

MR. BIERSTEKER: Tom, you want to read back 
the question? 

MR. HAMLIN: Wait a minute. Yeah, have her 
read it back. Go ahead, read it back. 

(The record was read by the reporter.) 

A. The use of propensity scores can indicate that, 
yes, but it's the — it's the use of them, it's not 
— it's not quite the way we use — use the word. 

MR. HAMLIN: Well let me mark this. 
(Plaintiff's Exhibit 3541 marked for 
identification.) 

Q. Sir, let me show you what has been marked as 
Exhibit 3541, which is a paper entitled "The central 
role of the propensity score in observational studies 
for causal effects" by Paul R. Rosenbaum and Donald 
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Rubin. Do you see that? 

A. Yes. 

Q. And this is an aut — or a paper that you 
authored with Dr. Rub — Rosenbaum? 

A. Correct. 

Q. Let me — and I believe the date of this paper 
is 1983; correct? 

A. Correct. 

Q. Let me direct your attention to page 48. 

A. Yes. I'm on 48. 

Q. All right. Section 3 you state in the first 
paragraph that "The general results we have presented 
suggest that, in practice, adjustment for the 
propensity score should be an important component of 
the analysis of observational studies because 
evidence of residual bias in the propensity score is 
evidence of potential bias in estimated treatment 
effects." Those are your words, along with Dr. 
Rosenbaum? 

A. Correct. 

Q. And that's a true statement? 

A. That's a true statement. 

Q. Now is it correct to say that propensity scores 
indicate a potential bias, but they don't give you 
evidence of actual bias? 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

348 

A. Comparing the distribution of propensity scores 
gives you an indication of potential bias, but it 
does — which does not necessarily convert to actual 
bias. 

Q. Now, would you agree that matched sampling is a 
standard technique in observational studies? 

A. It is a technique. Standard, yeah, it's — it's 
fairly standard. I like it. It's standard — I mean 
standard I guess is — I'm thinking is it common or 
not? It's a standard technique. There's no problem 
calling it standard technique. That's fine. I'm in 
a context now where I'm trying to be extremely 
careful about the uses of adjectives like that and 
I'm trying to think of is there a connotation to 
"standard" that might not be accurate, but I think 
that's fine, I think that standard is fine in this 
context. 

Q. All right. Now you have recommended matching 
based on estimated propensity scores; right? 

A. As one component of an analysis in certain 
situations, that can be very useful, yes. 

Q. Do you know of any studies published by the 
federal government that use matched sampling as a — 
or — Strike that. 

Do you know of any studies published by the — 
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the federal government that have used matching based 
on estimated propensity scores? 

A. I believe I do. I believe that was the NIH 
study that I had mentioned yesterday that involved 
Barry Graubard. I think that — when he talked to me 
I think it was matching these kids who were — 
infants who were exposed to the defective formula and 
matching them up with kids who were not exposed to 
the defective formula, and when he talked to me 
that's what I recommended using was estimated 
propensity scores to do matching, I believe. This is 
a long time ago, but I believe that's true. 

Q. Did — 

A. And I believe it got published that way. 

Q. You were not an author? 

A. No. As most things when I try to help people, 
just because I try to help them or consult on a 
project, I don't feel it's right to be an author. 

Q. Did you review the final results before they 
were published? 

A. No, I did not. 

Q. So as you sit here today do you have a 
recollection that in fact propensity scoring was used 
in that study? 

A. I have a recollection that it was used and I 
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have a recollection that it was used — used 
propensity scores. I also, if you want a government 
study, government-funded study, the one I was 
describing with the barbiturate exposure in utero I 
believe was funded by NIH and the work was conducted 
by people at the Kinsey Institute primarily, jointly 
with people in Copenhagen and that used propensity 
score matching. 

Q. And were you an author of that study? 

A. I was — there were pieces of work that came out 
of it so that pieces of work that were technical I 
was an author or coauthor of. There papers that Paul 
Rosenbaum and I did using that data set. There were 
papers that Neal Thomas and I published that used 
that data set to illustrate the idea. The real 
analysis from that data set, let me think. There 
were some publications that they did— the people at 
Kinsey did. Yes, I am a coauthor of one that used 
propensity score methods to match people up, and that 
appeared in a medical journal. What journal? I 
believe Journal of the American Medical Association, 
JAMA, I'm guessing 1995, 1996. I think there were 
four authors on it. If I looked at my vitae I could 
get the reference for you. Would you like me to do 
that. 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

351 

Q. Sure? 

A. You put me at the bottom. Okay. Let's see. 

Ah. It's number 205, 1995. The title is "In Utero 
Exposure to Phenobarbital and Intelligence Deficits 
in Adult Men" published in The Journal of the 
American Medical Association, Volume 274. It was 
joint with June Reinisch who through most of this 
period of time was director of the Kinsey Institute, 
S. Sanders is Stephanie Sanders, and E. Mortensen who 
is a Danish colleague. I don't remember what the E 
stands for. 

Q. Other than these two studies which you've 
identified, do you know of any other studies that 
have recommended or used matching based on estimated 
propensity scores? 

A. I could — I know of some. I just I can't 
recall them. If I had a copy — if you have a copy 
of 211, which is "Matching Using Estimated Propensity 
Scores: Relating Theory to Practice" published in 

Biometrics in 1996, I believe in the first paragraph 
we mention some. In fact I think there was one that 
Neal Thomas was involved with at Rand which was on 
health care costs. How about that? It's coming 
back. Champus Insurance programs, federal maybe was 
— I'm trying to remember now. I really wasn't 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

352 

involved in that study, except really peripherally, 
but as I remember it was a federal-funded study of 
two insurance programs used by the Armed Forces, and 
they had a new program that changed copayment sizes 
or something like that, they wanted to know what the 
effect was for the military, but it went country — 
region of the countries, I think it was implemented 
in California, and they had to look for bases, 
military bases in other parts of the country where 
they had the old programs, and they wanted to know 
what the costs to the federal government were of the 
new program versus the old program. And my memory is 
that what they did is they did — Rand did propensity 
score matching of the bases in California that would 
have treated to try to find control bases that were 
similar in terms of rank of the people who were 
there, the portion married, because that makes a 
difference if they have kids or not for medical 
costs, maybe racial composition, things like that, I 
believe. So I was really peripherally involved and 
just conversations with people. But I believe if I 
could grab that article I could probably grab the 
reference to it. I think it's — but it may not be. 

A phone call to Neal Thomas would work though to get 
that. Should I sit here and try to think of more 
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that might be in my — that I have been peripherally 
involved in? 

Q. Well would you agree that this matching based on 
estimated propensity scores is highly theoretical and 
there's been only a limited amount of statistical 
work evaluating this practice? 

A. No. I would agree that there's a tremendous 
amount of theory behind it that's — that supports 
its utility, that some of the theory is relatively 
deep, but the implication is that it's not an applied 
tool and it certainly is an applied tool. Well, I'll 
give you another example. Good. It involves the one 
on — with someone at the Harvard Medical School 
right now, Glen Chertow, which is the effectiveness 
of transfusions as a way to enhance the renal 
transplants, that there's some evidence that he has 
from large databases that people who have had 
transfusions have a better chance of not rejecting 
kidney transplants, and so for about a couple years 
we have been involved in doing propensity score 
analysis of — of that database for — for him. And 
also the study that I'm involved in with Richard 
Frank on health care costs related to different 
psychiatric programs uses propensity scores to es — 
to get them. I've been invited several times to 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

354 

attend dissertations of people with Healthcare Policy 
at Harvard Medical School who are using propensity 
score methods. In fact I'm supposed to go there I 
think in a few weeks, I've got an e-mail to respond 
to to do that. There's a study done in Economics 
comparing the effects of job training programs and 
observational studies where it's extremely effective 
duplicating the answers from a randomized experiment 
in a setting where people thought it was impossible 
to duplicate the results of a randomized experiment. 
It's applied frequently, more and moreso, as people 
have particularly become aware of it. I was — I was 
asked to write an article, invited article, for the 
Annals of Internal Medicine on exactly that topic 
because there's interest in it. I could go on, but I 
should probably stop. It's a long way of saying no. 

MR. HAMLIN: Okay. Let's mark this. 

A. Ten years ago the answer might have been yes. 

But as we sit here today the answer is no. 

(Plaintiffs' Exhibit 3542 marked for 
identification.) 

Q. I want to show you what's been marked as Exhibit 
3542 . 

MR. BIERSTEKER: Thanks. 

Q. This is an article entitled "Matching Using 
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1 Estimated Propensity Scores: Relating Theory to 

2 Practice" and you are an author along with Neal 

3 Thomas; right? 


4 

A. 

Correct. 


5 

Q. 

And the date of this article is 

March of 1996; 

6 

correct? 


7 

A. 

Correct. 


8 

Q. 

It was published in Biometrics; 

is that right? 

9 

A. 

Correct. 


10 

Q. 

This was a peer-reviewed journal 

; is that right 

11 

A. 

Yes, it is. 


12 

Q. 

Let me direct your attention to 

page 250. 

13 

A. 

Okay. 


14 

Q. 

All right. And the paragraph — 

first full 


15 paragraph on that page that begins "In typical 

16 matched settings..." 

17 A. Correct. 

18 Q. You go on to — to state that a better method in 

19 such cases essentially using an estimated propensity 

20 score as a primary matching variable; correct? 

21 A. Let me see. I'm trying to find exactly where 

22 you are. Better — better than what? Make sure I 

23 understand what the sentence is. Okay. A better 

24 method, fine. 

25 Q. And the method you're talking about is using an 
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estimated propensity score as a primary matching 
variable; right? 

A. Correct. It's not the sole one, but the primary 
one; correct. 

Q. And you go down to the last paragraph on the — 
on that page, page 250. 

A. Right. Let me just check one thing here. The 
reference is actually — these are — I'm glad, this 
is the article I was referring to that has — has 
these other references. The Aiken, Smith, and Lake 
paper, 1994, is a nursing home one where I think it 
used estimated propensity scores in medical care. I 
believe it did. I believe the Kane reference does, 
also in nursing homes, Kane, et al. I believe that's 
Journal of the American Geriatric Society. I think 
these are references to articles that actually use 
estimated propensity scores as the primary matching 
variables. I think that's what I think they are. I 
might be wrong, but I believe they are. Mental 
retardation, American Journal of Mental Retardation. 
So I think those are other examples. I guess I don't 
have the Champus or whatever that one I was thinking 
of may not be in here because I don't see anything to 
a Rand report. I think we had that in here but they 
wanted us to take it out because it wasn't in a real 
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journal publication, but it was a Rand report and — 
and I think the editor wanted it cut because of that, 
so I just — that's my memory, I might not be right, 
but I think that's right. I'm sorry. 

Q. Okay. Are you done? 

A. I apologize. 

Q. Now you're reminding a method here of an 
estimated propensity score to be used as a primary 
matching variable; right? 

A. Correct. 

Q. And that's the subject matter of your article; 
right? 

A. It concerns the — what happens when you do 
that, that's correct. 

Q. Now in the last paragraph on page 250 you write 
there has been only limited statistical work 
evaluating this practice beyond the theoretical work 
in Rubin and Thomas and some simulations in Drake and 
Gu and Rosenbaum. That's your statement; right? 

A. That's correct. 

Q. You're talking about the technique of using an 
estimated propensity score as a primary matching 
variable; is that right? 

A. What I meant — 

Q. Is that the technique you're talking about? 
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A. That's what I'm talking about and what I mean by 
statistical work is technical statistical work, not 
applications of it, because it's evaluating this — 
this practice, so limited statistical work evaluating 
this practice, that means the technical work 
evaluating the practice; correct. That's clear I 
think because it says "beyond the theoretical work 
in" and there are "some simulations". 

Q. And this article was published in March of 1996; 
right? 

A. Correct. 

Q. And that statement was true in March of 1996; 
right? 

A. It's true, but I — if you — I'll tell you what 
it — what it means if you want me to. 

Q. You don't — 

I'm asking you if it was a true statement in 
March of 1996? 

A. It's true when — when read properly. And if 
you want me to tell you what it — what it means, I 
can — 

MR. BIERSTEKER: No, that's fine. 

A. Okay, I'm sorry. It's true. 

Q. Now as of the time of this article, would you 
agree that very few scholars had published any 
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material on propensity scores? 

A. Okay. Are you referring to people doing 
technical work in a theoretical kind of way on 
propensity scores or do you mean scholars in other 
fields who use the techniques? 

Q. Both. 

A. Well, there — there are a limited number of 
people who knew the formulas, primarily Paul 
Rosenbaum, me, Neal Thomas, and some other people 
such as I guess Drake. There have been some more — 
since this article have been other things that people 
have — other people have done, I believe. But in 
the other sense of scholars, scholars in a medical 
area who have — who have used the technique, I refer 
to the GAO report, refer to the references of people 
— of people who have used the technique, refer to 
the Graubard thing, refer to the Ros — refer to the 
Reinisch, other publications that they I think have 
on the data set that we created by using this, 
created by matched samplings in estimating propensity 
scores, and — so I'm not fully aware of all of those 
articles, but I think — no, in fact I do know a 
whole collection more. 

There's an article that I was asked to write for 
the Annals of Internal Medicine on propensity scores 
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because it was becoming commonly used in medicine and 
they basically wanted an article by somebody who 
developed it, and in that article I have a collection 
of papers in medical journals that use propensity 
scores, some of which do — do some kind of technical 
exploration as well, and writing programs, software 
for doing it I believe is in there, as well as actual 
applications of it. If you have a copy of that 
article, I think that was one of the attachments to 
one of my statements, maybe not here, but there was 
that article to appear in the Annals of Internal 
Medicine, which I just saw page proofs of about a 
week ago, so it probably is out by now, but that has 
a collection, so I guess I don't agree with that, 
because of those — those articles. I assume they're 
scholars; they publish things in peer-reviewed 
journals. 

Q. Now you have suggested applying propensity 
scoring to survey contexts; right? 

A. In a much more limited way I have. The main 
promotor of that idea in surveys has really been Rod 
Little, a colleague I write with. I may have 
suggested it, and I — I have done it in certain 
contexts, but my focus on propensity scores has been 
much more in observational studies, so I'm just 
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trying — trying to be accurate here, so I — if I've 
— I'm sure I've suggested the use of it in certain 
cases, but my primary use has been in — in 
observational studies. Actually there's another 
collection of examples; I believe has been used at 
ETS to do comparisons of like different categories of 
people adjusting for collections of background 
variables as well, and I could — I know who to call 
to recover those, if it would be of interest. 

MR. HAMLIN: Let's mark this. 

(Plaintiffs' Exhibit 3543 marked for 
identification.) 

(Recess from 11:42 to 11:55 a.m.) 

BY MR. HAMLIN: 

Q. Professor Rubin, I show you what has been marked 
as Exhibit 3543. That's an article entitled 
"Projecting from Advanced Data Using Propensity 
Modeling: An Application to Income and Tax 

Statistics." I believe you were one of the authors; 
right? 

A. Correct. 

Q. This is published in the Journal of Business & 
Economic Statistics in April of 1992; right? 

A. Right. 

Q. Could you turn to page 2 or the second page, 
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which I believe is page 118. 

A. Uh-huh. 

Q. Now in the first full paragraph, the last 
sentence you indicate that you and others have 
suggested applying the technique of propensity 
scoring to survey contexts; right? 

A. This is in the first full paragraph? 

Q. First full paragraph. 

A. And I guess — 

Q. The last sentence. 

A. — the last sentence. Okay. Let me — 
observational studies, da, da, da, da. David, 

Little, Samuhel, and Triest (1983), Little, Little, 
is that the point? Yes. And as I said before, I 
think Rod Little was the — was more active in doing 
these, you know, surveys than I have been, I have 
focused on observational studies, although we have 
done work together using it in surveys. So my memory 
of the David, Little, Samuhel, Triest article was 
that he used it to — in a nonresponse setting to 
impute, yeah, that's right. Okay. I'm with you, 
yes. 

Q. All right. So you have in this article, along 
with others, suggested applying the technique to 
survey contexts; correct? 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




CONFIDENTIAL 


363 

1 A. Correct. 

2 Q. Now does NMES use propensity scoring? 

3 A. I certainly don't think so. 

4 Q. What about the Behavioral — 

5 A. Well — 

6 Q. — Risk Factor Survey? 

7 A. — I'm trying to think of there's a way to — 

8 well, no, not the people who did the survey I don't 

9 believe, although I don't know. I doubt it. 

10 Q. What about the Behavioral Risk Factor Survey, 

11 does that use propensity score? 

12 A. You mean In the way — the people who created 

13 the databases or are you talking about the analysis 

14 that Zeger, et al. did? 


15 

Q. 

No, 

I 

'm talking about the databases. 

16 

A. 

As 

it 

came to — but you said do they do 


17 propensity scoring. 

18 Q. Okay. Does — do the authors of the Behavioral 

19 Risk Factor Survey do propensity scoring? 

20 A. For that — 

21 MR. BIERSTEKER: Object to the form. 

22 A. For that survey is what you mean? I don't know 

23 whether they do or not. I haven't seen any 

24 information that would suggest that they do. 

25 Q. Do the researchers responsible for the Current 
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Population Survey do propensity scoring in that 
survey? 

A. I don't know. 

Q. Do you know of any survey conducted by the 
United States government that uses propensity 
scoring? 

A. I believe some do. This paper was done for the 
IRS, Internal Revenue Service. It addresses the 
problem that when people file tax returns they're not 
on time sometimes and they get — people get 
extensions, and of the people who get extensions — I 
don't know whether this is corporation or individual 
tax returns, it may have been for corporations — the 
more complicated returns tend to be later, the ones 
with more schedules attached, and Statistics of 
Income Division which funded this work requires — is 
required to make projections of what the tax 
structure will look like at the end of the full year 
which they report to the Office of Tax Analysis, 
which reports to Congress for tax change laws. And 
so the issue is, well, if we have to close out the 
data early and report to the Office of Tax Analysis 
before all the returns are in, how do we make an 
adjustment for the fact that the early returns are 
different from the late returns? And so this was a 
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technology to do — to do that adjustment because the 
kind of regression models weren't working at all. 

Q. Well in fact your article states that you 
propose an use of propensity score methodology? 

A. Correct. 

Q. Right? Now was this actually implemented by the 
IRS? 

A. Well, it was implemented in the context of this 
study. Do I know whether they've continued to use it 
today? I don't — I don't know — 

Q. Okay. 

A. — whether they have continued to use it today. 
Q. Do you know of any government survey that uses 
propensity score methods? 

MR. BIERSTEKER: Object to the form. 

A. I — well, I believe so. You have to interpret 
propensity score methods the right way. But if it's 
getting the proper — for unit nonresponse, the 
probability of being a respondent or not can be 
viewed as a — as a propensity score, as a function 
of certain characteristics of people, and I believe 
— I may be wrong — I believe that for — in some 
aspects of handling unit nonresponse in NHANES they 
did do — they did that and maybe they do — I don't 
know whether they do it now or not, but they actually 
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were trying to do that through some other kind of 
nonresponse as well. I think StatCanada uses things 
like that for some of them. I could sit here and try 
to think of the specific examples, if that would be 
helpful but — 

Q. And understand that what I'm asking you is do 
any of these government surveys use your method of 
propensity scoring? 

A. I understand. In — in their surveys — 

Q. Correct. 

A. — you're asking me? I don't know for a fact — 
for a fact, but I could — it's one of those things I 
could find out with a few phone calls. It wouldn't 
surprise me at all if they do. 

Q. But as you sit here today you can't think of 
any? 

A. As I sit here today I cannot think of an example 
where I know that they're doing it today; correct. 

Q. Now. Would you agree that there is a lack of 
good software for doing propensity scores? 

A. Yes, in the sense that there's not a piece of — 
of software with all of the bells and whistles you'd 
like to have for doing all the diagnostics that are 
associated with it and for handling missing data that 
may occur in the background variables as well, 
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there's not a complete software, although in simple 
form it's just a logist — logistic regression or a 
probit regression, so certainly the kind of software, 
for example, that Zeger, et al. had available to them 
is completely — the simplest versions of that can be 
used to make propensity scores, so they have the 
software available, for example. Most people do have 
the software available. It's not packaged in a way 
that has all the nice diagnostics that I think should 
be attached to it to make it a propensity score 
program as such, but they certainly have the 
computing, most people have the computing available 
to calculate propensity scores. It's quite simple. 

Q. So the answer to my question is yes, there is a 
lack of good software for doing propensity scores; 
right? 

MR. BIERSTEKER: Objection. 

A. There's a lack of software with all the bells 
and whistles that I think would be nice to — nice to 
have, so that somebody who doesn't know how to use 
statistical software can just push a button and get 
it, so it's like a doctor, I mean you have to — it 
depends upon the context in which we're talking 
about. If it's the context of users who are not 
statisticians, there is a lack of readily-available 
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simple-to-use software. If you're talking about 
statisticians or econometricians or epidemiologists 
who use probit regressions or logit regressions, the 
answer is no. It depends upon the context of the 
users. 

Q. Okay. 

A. Now I'm sure you can find that sentence in an 
article, and I can try to tell you what context that 
article was in, and I would hope that I wrote it in 
the context where I was referring to applied users 
and not matter mathematical statisticians or — or 
academic biostatisticians or academic 
epidemiologists. For them the software has been 
available for a long time. 

Q. All right. Do you recall giving your deposition 
in the Florida case? 

A. I recall that I gave a deposition in the Florida 
case, yes. 

Q. Yeah, July 9th, 1997? 

A. Yes. 

Q. And do you recall testifying that software for 
doing huge regression analyses has been around for a 
quarter century or more? 

A. Yes. 

Q. Do you recall saying that: There is a lack of 
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good software available for doing propensity score 
things. My impression is that it is starting to 
change? 

A. Correct. And that was in response, I believe, 
to a question, maybe you should just read the 
question, but I think it was in response to a 
question about why don't more people use propensity 
scored methods rather than regression methods? Is 
that correct? 

Q. Well I'm just asking you if you recall that 
testimony? 

A. Well I recall the testimony I think attached to 
that question, but it would help me if I could see 
what you're looking at. 

Q. Sure. 

A. Okay. 

Q. Let me just direct you to pages 106 and 107 of 
your deposition. 

A. You see, taken out of context it's hard to know 
whether I'm recalling it properly or not. 

Q. Okay. 

(Witness reviewing document.) 

A. Yes, the question is if what we're saying is 
correct and universally recognized, why do we have 
reports that are being published; for example, by the 
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Government Accounting Office in the form that we have 
them? Let's just take the breast issue, what was 
published in that format. And — the question — I 
was — yeah, so the context is users, 
non-statisticians who — who — not professional 
statisticians, and for them there's a lack of good 
software. The GAO report, the primary person 
involved with it was somebody named Judy Droicourt — 
it's going to be a tough smelling, I'm not sure I can 
remember exactly — but she did not have a Ph.D. in 
statistics, I don't think she even had a Master's 
degree in statistics, but she was in charge of this 
project trying to do that. It would have been nice 
to have point a point and click software for doing 
propensity score methods, and that's the context in 
which I answered that question. 

Q. Now if we go back to your report, which is 
Exhibit 3540, I'd like you to turn to page 10. 

A. Okay. 

Q. Now at the top of page 10 you write that 
"Without the propensity score analysis, I would have 
no confidence in the reliability of the results of 
the regression analysis." 

Do you see that? 

A. Yes, I do. 
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Q. And you're referring there to the Zeger, et al. 
report; right? 

A. Correct. 

Q. Now I want you to assume that the 
epidemiological studies cited by the Surgeon General 
in his 1989 report do not use propensity scores. Is 
it your testimony that you would have no confidence 
in those results? 

A. I have not read the report, so I'm not sure how 
I should have confidence or not in the report. I 
mean generally I would have — I will believe that a 
report was written by competent people and that what 
they're saying there they believe. Does the fact 
that they — that the report probably did not, almost 
certainly did not use propensity score methods, does 
that fact alone mean I have no confidence in the 
results just because of that alone? No. 

Q. It does not mean — 

A. It does not mean, double negative. 

Q. — that you have no confidence? 

A. You have a double negative. Double negative. 

It does not mean I have no confidence. It does not 
mean I have confidence necessarily either, but it 
doesn't mean I have no confidence. 

Q. Now have you — do you know of any studies in 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

372 

the epidemiology of smoking and disease that do use 
propensity scores? 

A. Smoking and disease. To address that I'd 
probably have to look at that list of references that 
I have in my Annals of Internal Medicine article to 
see if any of them address that. I don't know how 
complete that list is. But as we sit here today I 
cannot identify a particular article that does. 

Q. Have you yourself ever attempted to calculate 
propensity scores for a study in the epidemiology of 
smoking and disease? 

A. Yes. 

Q. Other than this case? 

A. No. 

Q. I want you to assume that the health expenditure 
studies on the cost of smoking-related illnesses do 
not use any propensity scoring. 

A. So — 

Q. Does that fact alone cause you to not have any 
confidence in those studies? 

A. Not necessarily, but I — it talks about all 
studies, I don't even know whether — what questions 
they're addressing. Are they — should I assume that 
they're addressing the question — the descriptive 
question of what the SAFs are in — in different 
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cells of people; say age category, sex, seatbelt use, 
education cell, when comparing smoking and nonsmokers 

— smokers and nonsmokers to get SAFs in that cell. 
Does the fact they don't use propensity scores lead 
me to have no confidence in the results? Given what 
I've seen in the Zeger report and the initial 
propensity scores work I've done on NMES, the answer 
would be yes, I don't have much confidence in those. 
Q. Now would the answer be the same for studies 
such as the Framingham — 

A. Yes. 

Q. — which is Exhibit — 

A. Yes. 

Q. — 3536? 

A. Yes. 

Q. All right. 

A. Because when we looked at this yesterday I was 
looking for a table that would give me any indication 
of how far these two groups — how far away these two 
groups start with respect to their — their 
background variables, and there's no table that 
indicates how far apart they are, and so I might have 
some — some concerns about that, and so therefore I 

— I don't have much confidence until I saw that 
kind of analysis, and — to show how far apart these 
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two groups started, because they're using linear 
models to do the adjustment, and if the groups are 
far apart I don't have confidence in linear models. 
I'm not saying it would make a difference, I'm not 
saying it would not make a difference, I just don't 
know, and I don't think the authors know either. 

Q. So you don't know whether propensity scoring 
would make a difference in The Framingham Study or 
not; right? 

A. I don't know whether it would but I — but 
because I don't — there's no information about how 
far apart the smokers and nonsmokers started for this 
conclusion that's being reached, I don't have 
confidence in — in the results of — that the 
results of these adjustments which are basically 
bringing people into the same cell actually are 
reliable. 

Q. Okay. And you don't know what difference 
propensity scoring would make in the results of the 
Zeger, et al. report; right? 

A. I don't know what difference it would make, but 
I — but I do know that it's — it's an issue in that 
report, because the groups do start out within age 
and sex cells pretty far apart, farther apart than 
you can rely on — on models to — to take care of 
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unless you really do an extremely careful job and — 
and look for these kinds of problems, which 
apparently was not done. 

Q. All right. My question was: You don't know 
whether applying propensity scores to the Zeger model 
would make any difference in the results; right? 

A. In the point estimates, no, due to it would make 
a difference in the standard errors and the 
confidence intervals would increase because you have 
to account for the fact that — that you're doing 
these extrapolations and the models can't — can't be 
relied on. 

Q. And have you applied propensity scores to the 
Zeger models to determine what the confidence 
intervals are? 

A. No. 

Q. So as you sit here today you really have no 
specific knowledge as to what effect applying 
propensity scores would have on the Zeger model; 
right? 

MR. BIERSTEKER: Objection; asked and 
answered and mischaracterizes the testimony. 

THE WITNESS: I agree with that. 

A. Because the initial propensity analyses I've 
done indicate that the smokers and nonsmokers start 
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out fairly far apart even within age and sex cells, a 
consequence of that is I don't know whether the point 
estimates of SAFs would have changed, but because 
they start out far apart it means that extrapolation 
is being done which leads to larger standard errors, 
if properly accounted for, than the standard errors 
that are found assuming the model is correct, so I do 
know that adjusting for propensity scores and using 
that — the information that comes from that would 
lead to larger standard errors. For the confidence 
intervals to be at all honest confidence intervals, 
they would have to have been bigger than the 
confidence intervals that they would get assuming the 
model is true. 

Q. So you're saying that the confidence intervals 
would be larger using propensity scores? 

A. Yes, if you did a proper allowance for the fact 
that the — that there's extrapolation involved 
because the groups start out fairly far apart, the 
smokers and nonsmokers start out fairly far apart, 
that you have to allow for the fact that the model 
that you're using has been asked to do too much, and 
that therefore leads to broader intervals to allow 
for the fact that the model may not be exactly right. 
Q. So that the range of point estimates or 
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statistical answers would be larger using propensity 
scores; right? 

A. Yes. It — 

Q. Now let me ask you this: On page 10 — 

A. Well let me say, that — just to clarify that. 
That would be the expectation, so in — in a 
statistical sense that's the right answer. Do I know 
each one of them would be? I don't know for a fact 
each one of them would be bigger, although if 
properly done they probably would be. I'd have to 
think about, whether I believe that would be true 
just generally statistically or be true for each 
one. Let me think about it for a second. I believe 
that would be true for all the confidence intervals; 
confidence levels would be bigger for all of them. I 
think so. 

Q. On page 10 you write that "The implicit 
assumptions underlying the Zeger Report's regressions 
do not appear to be" — "do not appear to have been 
examined for plausibility." What assumptions are you 
talking about? 

A. Well the various assumptions about — in the 
expenditure part of the model about linearity. I 
don't — I didn't see any exploration in the report 
at least of discussion looking for interactions or — 
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or product terms, power — the power, transformations 
in the predictor variables, the kind of assumptions 
underlying the mills ratio adjustments rely on 
normality, bivariate normality; I didn't see any 
study of that. The probit models generally also have 
that issue. I didn't see any exploration of what was 
done in exploring for non-linear terms in — in those 
models either. In the — this was written in June 
19th on the Zeger report as the report existed at 
that time. And from reading the depositions that's 
been a moving target, so I'm not sure what state it 
is now. 

MR. HAMLIN: Well, I move to strike that 
last sentence. 

BY MR. HAMLIN: 

Q. I asked you about the implicit assumptions that 
you were referring to in this text. Now, have you — 
A. Oh. 

Q. — given me a list of those implicit assumptions 
that you are referring to? 

A. I've given you a partial list of those. 

Q. I want a complete list of the implicit 
assumptions that you're making reference to in this 
text? 

A. I don't have a complete list in my — in my head 
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right now. It would take me time to produce a 
complete list. It's difficult from the report 
because the report is — is vague and I believe even 
incorrect in some places when it describes things 
that its doing, so I'm not sure about that, but it's 
difficult to — to make a complete list. I'd have to 
go back to looking at the code to see what was 
actually done. And so I can't give you a complete 
list right now. I gave you indications and examples 
of the types of things. 

Q. All right. And have you determined whether any 
of these assumptions are wrong? 

A. Well they're probably all wrong. That doesn't 
mean they're — they're — have tremendous 
consequences. 

Q. Okay. So you don't know whether there would be 
any difference in the results even if those 
assumptions were changed; correct? 

A. Correct. Well again there — there's one 
question concerning point estimates in the results 
and one concerning confidence levels in the results. 
The results would change, and in fact the confidence 
levels would be bigger if — if it were done more 
appropriate I believe. 

Q. And you're referring there to propensity scores? 
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A. Or even if you explored different models you 
might find just answers changing and so a proper 
interpretation of what a confidence level would be 
would include the fact that the answers can change 
depending upon the models. 

Q. Have you attempted to determine whether the 
answers change depending upon the model? 

A. No, I have not. 

Q. And so you don't know what result would — 
Strike that. 

You don't know what the result would be by 
changing the models? 

A. For the point estimates, no, I don't. 

Q. You don't know if it would change them at all; 
right? 

A. Well I'd be shocked if they were the same to — 
same position. But would they change them in a 
substantial way? I don't know. 

Q. Now Roman numeral IV you state that "The Zeger 
Report fails to adhere to sound statistical methods 
to address missing data problems, rendering the 
estimates of smoking-attributable health care costs 
unreliable." Have you corrected those problems and 
run the model? 

A. No, I have not. 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

381 

Q. Now suppose you corrected the missing data 
problems. What effect would it have on the Zeger, et 
al. estimate of damages? 

A. Well one thing that wouid — I don't know what 
it would do to the point estimates, but again the 
interval estimates would be increased probably 
substantially to reflect the uncertainty in the 
imputations. 

Q. And right now that's based on your speculation, 
not on actual work that you've done; right? 

A. The fact that the confidence intervals would 
grow. Well, I — well it's based on the following 
thing: If I give you one number here, now I give you 

two numbers that are different, is there more 
variability in the two numbers that are different 
than the one number that's here? So it's got to be 
more variability? So the confidence levels have got 
to go up. Have I done it to show you how much it's 
going to go up? No. In lots of examples on other 
data sets, yes, theoretically it has to go up, 
practically it has to go up, it goes up. It would go 
up here too. 

Q. Do you have any plans to do it? 

A. Not now. That means not at present. 

Q. Okay. Let me direct your attention to page 11, 
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and specifically I want to point you to the last 
sentence in that first full paragraph that states 
"The treatments of these missing data problems by 
the Zeger Report does not adhere to sound statistical 
methods." 

Do you see that? 

A. Yes. 

Q. Now would you agree that there are three general 
strategies for analyzing missing or incomplete data, 
the first one being imputation; correct? 

A. That's a general strategy; correct. 

Q. The second general strategy is weighting; 
correct? 

A. That is a second general strategy; correct. 

Q. And then the third general strategy would be the 
direct analysis of incomplete data which analyzes all 
data using approaches like maximum likelihood? 

A. Those are the three approaches that have general 
validity to them; correct. Or can have general 
validity, properly done. 

Q. Does NMES use any one of those general 
strategies to deal with missing data? 

A. You mean the — the authors of NMES? 

Q. Yes. 

A. Yes. No, they don't. Not that I'm aware of. 
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Q. And do the authors of BRFSS use any one of those 
strategies to deal with missing data? 

A. Not that I'm aware of. 

Q. I think we previously agreed that multiple 
imputation — Strike that — that imputation can take 
a number of forms; right? 

A. Yes. 

Q. And that one of those forms was filling in a 
value predicted from modeling the missing variable 
given observed variables using respondent data; 
right? 

A. That's an aspect of creating imputation. That 
can be an aspect of creating imputation. 

Q. So regression can be used to predict missing 
values; right? 

A. It can be used to predict missing values. That 
doesn't mean that if you do it and put in the 
predicted value, it's a valid method. I'm just 
saying it can be done. That doesn't mean it should 
be done. 

Q. But it might be regarded as reasonable; right? 

A. Not if you just put in the predicted value, no. 
It might by some, but it's not, it doesn't work. 

Q. Okay. Can you turn to page 3534. Or strike 
that — Exhibit 3534. 
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A. 3534. 

Q. That's your article entitled "Multiple 
Imputation in Health-Care Databases: An Overview in 
Some Applications". 

A. Here it is. Okay. 

Q. That's the article entitled "Multiple Imputation 
in Health-Care Databases: An Overview in Some 
Applications" authored by you and Nathaniel 
Schenker. 

A. Correct. 

Q. And in that first paragraph. Introduction, you 
write, "Many different procedures have been proposed 
for imputation;" — right? 

A. I'm with you. 

Q. "for instance," you go on to say, "filling in 
the respondents' mean for the missing variable or 
filling in a value predicted from modeling the 
missing variable given observed variables using 
respondent data..."; correct? 

A. Correct. 

Q. And then you give a specific example? 

A. Right. 

Q. And you say that "when the missing value is 
personal income, a linear regression model predicting 
log (income) from demographic characteristics such as 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

385 

age, sex, education, and occupation might be regarded 
as reasonable"; right? 

A. Right, might be regarded as reasonable. 

Q. Right. 

A. So that by — that was setting people up to 
think, gee, it might sound reasonable but in fact it 
doesn't work well. That was the context of it. 

Q. And in fact researchers today do use that 
method; correct? 

A. Some may. 

Q. In the past researchers and statisticians have 

used that method; right? 

A. Some have done so. 

Q. Have you ever used that method? 

A. And just plugged in a value like that? If I 
have, it would have been a very long time ago and it 
would have been only in the context, I believe, of 
exploratory analyses or in a context where the number 
of missing values, fraction of missing information is 
so slight that it wouldn't have made a difference. 

Not where most of the data are missing, and nor in a 
— in an analysis that I would regard as something 
that important decisions were being made on. 

Q. So the answer to my question would be that you 
might have used that method in a limited way? 
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1 A. I — I don't believe I — I have in anything 

2 that's ever been published or that I want to be 

3 published, but it's possible a very long time ago 

4 that in some data set somewhere where there is very 

5 limited missing data I may have done something like 

6 that, but I — I doubt it. And I should be ashamed 

7 of myself if I did. 

8 Q. Now would you agree that Zeger, et al. made use 

9 of regression to predict missing values in the 

10 Plaintiffs' damage model? 

11 A. Yes. 

12 Q. Would you agree that Zeger, et al. also employed 

13 maximum likelihood in the probit models to predict 

14 whether someone is a smoker given they have a 

15 disease? 

16 A. Yes. 

17 Q. Now let's go back to your report, specifically 

18 page 11, where you identify a number of general types 

19 of errors, as you characterize them. Do you see 

20 that? 

21 A. Yes, I do. 

22 Q. Specifically I think you say that "The 

23 imputations in the data sets used by the Zeger Report 

24 reflect several general types of errors." That's at 

25 the top of the second full paragraph. 
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Do you see that? 

A. Correct. 

Q. And then you describe those errors? 

A. Correct. 

Q. Now the first one you talk about is best 
predicted values. And I think you identify BRFSS and 
you state that medical expenditures are nowhere 
observed, they are imputed on NMES; is that — is 
that right? 

A. Correct. 

Q. Do you know where Zeger does this in — in the 
model? 

A. I'm not sure what the question means. 

Q. Okay. I believe you say that one of the errors 
in the Zeger report is this use of best predicted 
values; right? 

A. Correct, the — the use of best predicted values 
in general is an error. 

Q. Right. And then you give an example. You say 
"In BRFSS," — "BRFSS, medical expenditure data are 
nowhere observed and are imputed, using best 
predicted values based on NMES models." Right? 

A. Correct. 

Q. Now is that something that Zeger, et al. do? 

A. They effectively do that. 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

388 

Q. And do you know in which model they do that? 

A. I'm not sure what "which model" refers to. 

Q. Well let's talk about the disease model, and by 
that I mean lung cancer, COPD, and coronary heart 
disease. 

A. I — in that model I don't believe they — wait 
a minute. Let me think for a minute. I think in the 
lung disease model they go from NMES with age, sex, 
and insurance type, I believe, straight to the 
claims, if I'm getting that right. So they might do 
that there, but I — I'd have to check — check notes 
because they do different things different places and 
the — the logic is a bit fuzzy to me at least of why 
they make alterations, so it's not something that's 
easy to keep in mind, someone else's what I regard as 
arbitrary switchings around, but I think that's 
correct, I think they — I think they go from NMES to 
claims for that disease. 

Q. Now is it your testimony that multiple 
imputation should be used to resolve this problem 
with best predicted values? 

A. It's my testimony that using best predicted 
values in general is — is incorrect. We've already 
discussed that there are other methods of validly 
handling missing data. I think that in many context 
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— contexts multiple imputation is a very effective 
way of doing that. So is it my testimony that 
multiple imputation had to have been used here to get 
valid answers? It's not. I'm not saying that. 

Q. Okay. So you're not saying that multiple 
imputation had to be used here to get valid answers; 
right? 

A. Correct, I'm saying that best predicted values 
are generally incorrect. 

Q. Now, have you used any imputation problem — 
Strike that. 

Have you used any imputation procedure to 
attempt to eliminate these best predicted values? 

MR. BIERSTEKER: I'll object to form. 

A. In — 

Q. In the Zeger model. 

A. In the Zeger data set model combination focused 
here, not in any other data set we've been talking 
about; — 

Q. Correct. 

A. — correct? No, I have not. 

Q. Do you know as you sit here today what 
difference — Strike that — whether the Plaintiffs' 
damage would increase or decrease if you used 
multiple imputation in place of these best predicted 
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values? 

A. Do I know whether the use of multiple imputation 
rather than best predicted values would have changed 
the numbers that come out of the Zeger model 
substantially? No, I do not know whether the point 
estimates would change substantially. Again the 
intervals would be bigger, interval estimates would 
be bigger because there'd be more uncertainty when 
you say I don't know the value of this number versus 
I — I know it's right here and that's exactly what 
it is when you don't know it. So I think I've 
answered that. 

Q. So you don't know what — what — whether it 
would have a substantial effect on the point 
estimates, but it may have an effect on the 
intervals? 

A. The intervals, I believe, would certainly get 
bigger, but I don't know whether these point 
estimates of SAFs and so forth would — would change 
in a substantial way. They would certainly change, I 
believe, but substantially, I don't know. Even 
multiple imputation though wouldn't help address the 
other question, the causal question, but it might 
help address the descriptive question in a more valid 
way. 
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Q. Let me ask you about this next problem you 
identified concerning improper conditioning. You 
state that "When imputing expenditures, weight, and 
seatbelt use in NMES, smoking is not used." 

Can you explain to me what you mean by that? 

A. Well what — when the — the NMES people, for 
example, imputed expenditures, I believe their 
hot-deck did not use smoking. I believe when Zeger, 
et al. imputed probabilities of weight and seatbelt 
use in NMES, smoking was not used as a predicted 
variable, nor was expenditures, I believe. That's 
what I mean by the sentence. 

Q. And is it your testimony that smoking should 
have been used as a predictor variable? 

A. I'd have you clarify "should" there, because 
"should" should not have the implication that doing 
that by itself would fix up the missing data problems 
in this context. I know we're not talking about the 
causal question, just the descriptive question, but 
when doing imputations you have to use all the 
information at the same time, all the variables have 
to be used to impute all the missing variables, not 
the way they're doing it here. This gets the wrong 
answer. It's inconsistent, badly — badly biased in 
general. 
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Q. Now have you actually done any imputations where 
you used smoking in the Zeger model? 

A. In this data set you mean? 

Q. Yes. 

A. No, I have not. 

Q. So you don't know what the results would be if 
you ran them? 

A. No, I do not. 

Q. And you don't know what effect, if any, it would 
have on the point estimates; right? 

A. That's correct, I do not know. 

Q. Now what values would you condition an 
imputation of expenditures on? 

A. Everything that's available. There's a — 
there's a rather large literature now which many 
people are aware of the authors of the report, on how 
to handle missing data problems in — in a general 
sort of way, how to do imputation, in a way that 
conditions on all the variables. 

Q. Now, in that next sentence you say, "To impute 
missing insurance information and seatbelt use in 
BRFSS for certain years, moreover, the so-called 
'hot-deck' method being used only conditions on a 
few variables, and even then with only limited 
success because of the collapsing rules being used." 
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Now what hot-deck method are you referring to? 

A. I'm referring to the — to the hot-deck that 
categorizes the variables that can be used to control 
for the imputation, and then form cells based on 
those categorical variables in which you hope to find 
a respondent and a nonrespondent. For each — sorry 
— in each cell where there's a nonrespondent you 
hope to find at least one respondent who is called a 
donor, who can donate his missing values to the 
nonrespondent. If the cell where the nonrespondent 
falls has no donors, has no respondents who match, 
then they have these collapsing rules where they 
either eliminate one of the variables that's used to 
form the tables, the cells, or they make the values 
of the variable coarser, so let's suppose, for 
example, to indicate coarsening, if one of the 
variables was age, I only had five categories of age 
and there is no matching person, we now might make it 
two categories of age and see if we had a matching 
person then. So that's the kind of hot-deck I'm 
referring to. 

Q. Are these variables in the hot-deck derived from 
BRFSS? 

MR. BIERSTEKER: Object to form. 

A. As opposed to — we're talking about BRFSS, so 
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they're in BRFSS. Were they all in BRFSS or were 
some of them imputed from somewhere else first? Is 
that the question? 

Q. Where did they come from? 

A. Well we're talking about BRFSS so they came from 
BRFSS. We're talking about BRFSS, I believe, so the 
respondent and nonrespondents in BRFSS with respect 
to the variable are trying to be matched up by 
variables that are — that both the respondents and 
nonrespondents have observed in BRFSS or have been 
treated as if they were observed in BRFSS. 

Q. Now you say that these variables are used to 
form cells; right? 

A. Correct. 

Q. Now the variables, as I understand it, come from 
BRFSS; is that — is that your testimony? 

A. We're talking about BRFSS here so they'd have to 
— we're talking about BRFSS. Let me — well wait a 
minute. Imputing — imputing into BRFSS. Into 
BRFSS. That would — they may be coming from — Oh, 
boy, is this coming from NMES maybe? I don't 
remember. Again it's such a — I'm trying to 
remember the details of — of someone else's 
procedures which are — have mistake after mistake 
after mistake and trying to remember the details of 
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this mistake compared to some other mistake. Whether 
it occurred in this data set or that data set is not 
that easy, especially when the report doesn't 
describe it and you have to read it from code. So do 
I know for a fact that the hot-deck cells are 
performed within BRFSS to be imputed to BRFSS, or are 
these hot-deck cells again using values from NMES 
instead? As I sit here right now I am rather 
confused. In fact actually I got more confused after 
I read the depositions of the people, but I'm 
somewhat — I don't know for a fact whether the 
hot-deck collapsing rules were created on BRFSS or 
NMES for the imputation of missing insurance and 
seatbelt use. I'd have to look in some notes in 
order to recover that, to be complete, to be 
completely correct. 

Q. And do you know whether this occurs in the 
disease model or the diminished health status model? 

MR. BIERSTEKER: Object to form. Do you 
want me to explain the objection or not? 

MR. HAMLIN: (Shakes head.) 

MR. BIERSTEKER: Okay. Go ahead. 

A. The data set is used in these models, the 
imputations are created in the data set, so if the 
data set is used in a model, then the imputations are 
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used in a model, so I don't think the question was 
quite properly formulated. Are you asking a question 
about whether BRFSS is used? 

Q. No, I'm asking where this so-called hot-deck 
method is used in the model? 

A. It's — the hot-deck method is used to create 
imputations. 

Q. And where are the — excuse me. Where are the 
cells used in the model? 

A. The cells of the hot-deck are used to create 
imputations. If you're talking about the model as to 
everything that they do, that's where it's used, the 
cells for the hot-deck are used to create imputations 
into the data set. 

Q. And are those cells used in all the models that 
Zeger uses, that Zeger employs? 

A. I don't think the question makes sense. At 
least I don't understand the question. 

Q. Well what I'm trying to get at, frankly, is what 
you mean — 

A. Okay. 

Q. — when you say the so-called hot-deck method 
being used only conditions on a few variables. Are 
there — do you know the variables that are being 
conditioned on? 
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A. I do not remember them right now, no. 

Q. Now have you looked at the Zeger, et al. model 

to determine what proper conditioning ought to have 
been done? 

MR. BIERSTEKER: I'll object. Asked and 

answered. 

A. In a general sort of way, yes, that all the 
databases that are being used should be collected 
together and there is a collection of missing values, 
some of it extreme, that cross big blocks of missing 
values across these data sets. The — the correct 

way to handle that is to impute all of it at once 

using all the information that's available in the 
data, thereby creating imputations that are 
internally consistent with each other and are 
efficient in the sense they use all the information, 
but to do that you're going to have to rely on — on 
some assumptions because of this same setting — 
setup as we had before where we were talking about 
the X, Y, Z example, and there's certain things that 
are not jointly observed. The problem with doing 
that, you're going to have make some assumptions 
about conditional independence or conditional 
relationships that the data don't address, so those 
assumptions you've made explicit and then all the 
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data should be imputed simultaneously. 

Q. But you haven't done that; correct? 

A. No, I have not. 

Q. Now let's go to the next problem. You 
characterize it as use of part of known data. Can 
you tell me what you mean by that? 

A. Sure. It's — what I was just describing was 
addressing both improper conditioning and part of the 
known data. The improper conditioning I would say 
even within a data set such as NMES that the 
imputations don't condition on all the variables that 
are available in NMES, so that's — if you think of 
the data matrix, the variables being columns and the 
units being rows, this is referring to the fact that 
in NMES, for instance, that some columns with missing 
values only use the information in some other columns 
and not in all the columns, so that was a comment 
about columns of the data set. This part of known 
data refers to going the other way. We stack all the 
data sets on top of each other for different people 
and when doing imputations in NMES you should use all 
the rows from the other part, all rows which are the 
units, so you should use all of that to create 
imputations everywhere, thereby creating a giant the 
data set with internally consistent relationships 
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rather than doing it piecemeal and in hoping somehow 
it all works out, which there's no reason why it 
should. 

Q. So are you saying in this section that all known 
data should be used? 

A. All the data that they're using, all their known 
data. I'm not saying go outside and get other data 
sources necessarily, but all of their data; NMES, 
BRFSS, claims, really should have been stacked on top 
of itself as units that are relevant with perhaps 
indicators of, if you want, from NMES, from Minnesota 
and otherwise. There are choices to be made, but 
they all should be dealt with as one large missing 
data problem and handled correctly and validly that 
way. 

Q. Now have you actually done that? 

A. Not in this data set, no. 

Q. So you don't know what effect on the estimates 
that would have; right? 

A. I do not know what effect that would have on the 
point estimates of the output of this model; 
correct. Again the intervals would no doubt, if you 
did it appropriately, widen because of the 
uncertainties that are involved, that are varied by 
the way it's done. 
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Q. But you haven't done that either? 

A. Correct, I have not done that. 

MR. BIERSTEKER: What time you want to 
break for lunch? Pretty soon? 

MR. HAMLIN: Yeah. 

MR. BIERSTEKER: It's almost 1:00. 

MR. HAMLIN: We can break now. 

MR. BIERSTEKER: It's all right if you have 
a couple things you want to clean up; if not, I don't 
think that you have to do it instantaneously. 

MR. HAMLIN: We'll go off the record. 
(Luncheon recess from 12:51 to 2:15 p.m.) 
AFTERNOON SESSION 
BY MR. HAMLIN: 

Q. Professor Rubin, could you turn to page 12 of 
your report. 

A. Okay. 

Q. You identify another problem with the Zeger, et 
al. model on page 12 as, quote, "No Account of 
Uncertainty in Estimations." Do you see that? 

A. Yes, I do. 

Q. You go on to say that because they did not take 
into account this uncertainty in estimations that 
this treatment can lead to biased estimation of final 
quantities and always leads to underestimation of 
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statistical uncertainty, unless subsequently 
addressed by special techniques, which do not appear 
to have been employed in the Zeger Report." 

Do you see that? 

A. Yes, I do. 

Q. What special techniques are you referring to? 

A. A — one example of another special — well, let 

me first say that in this context I was saying that 
if it hadn't — if they didn't reflect uncertainty of 
estimation as would have been done using multiple 
imputation; for example, then you — if you're doing 
it the way they did it, even if you put aside the 
question of biased estimation and you're just trying 
to get statistical uncertainty, the kind of special 
techniques that would be — could have been used 
would be jackknife or bootstrap methods where you 
re-drew the imputations on the jackknife or bootstrap 
samples, for instance. 

Q. Okay. So — 

A. And my understanding is that was not done. 

Q. Did you perform that procedure on the Zeger 
model? 

A. No. 

Q. Did you know whether performing that procedure 
on the Zeger model would change the Plaintiffs' 
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damages in this case substantially? 

A. I do not know whether it would change the point 
estimates of damages — well the point estimates of 
these quantities they calculate substantially, but I 
do know that the standard errors confidence intervals 
would have to go up, that the confidence would have 
to be wider and standard errors would have to be 
larger because there would be more uncertainty being 
reflected. 

Q. Since you haven't done it you don't know how 
much larger those standard deviations would be; 
right? 

A. No, I don't know exactly how much larger. Let 
me think about it for a moment. They get bigger by 
the fraction of missing information in the technical 
sense of missing information, and because in some 
instances there are large amounts of missing data — 
no, I don't know. I'm just going to stop there. I 
was trying to be overly detailed. 

Q. Let's go on to this next problem that you 
identified as "Reliance on Strong Unsubstantiated 
Assumptions". You say in the second sentence, "The 
analyses in the Zeger Report appear to force many 
coefficients in models to be zero, thereby assuming 
the absence of relationships between certain 
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variables. " 

Do you see that? 

A. Yes. 

Q. Have you run the Zeger, et al. model where the 
coefficients were not forced to zero? 

A. No, I have not. 

Q. And when you say that the coefficients were 
forced to zero, do you mean that the coefficients 
were eliminated because they were not statistically 
significant? 

A. That's one of the — one of the reasons, one of 
the ways in which they were forced to be zero. Other 
ways which they were forced to be zero, I believe 
there were just some — some ones that weren't 
included in certain relationships for other reasons 
or were left — interactions that were left out, for 
example. 

Q. Do you know of any expert in this case that has 
run the Zeger model allowing the coefficients to 
remain in, whether — whether statistically 
significant or not? 

A. Do I know of an expert? Well, from reading the 
depositions of — of the authors, my memory is that 
it appears that they — they have tried some things 
different ways back and forth. 
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Q. 

"They" 

being 

Zeger, 

et al.? 

A. 

"They" 

being 

Zeger, 

et al. And so if — if 


3 that's accurate, then they may have done things 

4 different ways and — but I don't think that there 

5 was any conclusion reached on the basis of that, that 

6 I remember from the deposition at least. 

IQ. Do you know of any expert for the tobacco 

8 companies that has engaged in that exercise? 

9 A. Where leaving the coefficients — no, I don't, I 

10 haven't read anything to that effect, and I don't 

11 think I've been told either. 

12 Q. Do you know what difference that exercise would 

13 make in the Plaintiffs' damages in this case? 

14 A. I don't know what difference that would make in 

15 the numbers that come out of the analysis. 

16 Q. Now you also indicate on page 11 that Zeger, et 

17 al. rely on strong bivariate normality assumptions; 

18 do you see that? 

19 A. Yes. 

20 Q. Would you agree that bivariate normality 

21 assumptions are commonly used by statisticians in 

22 creating models? 

23 A. Univariate normality assumptions are commonly 

24 used. Bivariate normality assumptions in these kinds 

25 of models where it's a bivariate outcome are less 
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commonly used, but to characterize it as common, 
yeah. Not uncommon at least. Not uncommonly used. 

Q. And have you used bivariate normality 
assumptions? 

A. Sure, yes. And certainly in many theoretical 
papers or methodological papers I have. To analyze 
data I try to be more cautious with using them to get 
answers, but I — I'm sure I have. 

Q. What assumptions — well, strike that. 

Have you run the Zeger, et al. model using any 
different assumptions regarding normality or 
distribution? 

A. No, I have not. 

Q. Do you know what effect running the Zeger, et 
al. model with different assumptions would make on 
the Plaintiffs' damages? 

A. I do not. 

Q. What assumptions do you think would be 
appropriate in these circumstances? 

A. To explore? As possible alternatives? 

Q. (Nods head.) 

A. To give a good answer to that question I really 
would have to see the data and — and do some 
preliminary analyses, but the kinds of things I would 
have in mind would be potentially using 
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longer-tailed, that's longer-t-a-i-l-e-d, 
distributions that have the same kind of symmetry 
properties as the normal does, as well as considering 
other kinds of transformations to reflect 
asymmetries. 

Q. You also say that Zeger, et al. make conditional 
independence assumptions across expenditure 
categories. Can you tell me what you mean by that? 

A. Let me recover that. Yes, I think — let's 
see. I think that's related to the fact that the 
models for the expenditure categories are all 
estimated independently, so they're essentially 
assuming that the expenses, given predictor 
variables, are independent because the — there's no 
— because they're being done separately, and 
because of the way that some coefficients are being 
forced to be zero, you could get different answers if 
you didn't force in that conditional independence. 

Q. What would you do differently? 

A. I would go back and try — well let me — to — 

I would go back and try to address the question 
that's being asked on the data set and not do any of 
this . 

Q. Okay. But my question was more specific. What 
would you do differently from making conditional 
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independence assumptions across expenditure 
categories? 

A. In order to address what question? 

Q. In order to address the — the issue that you 

raise in your report. 

A. It's a little like saying here's a — they're 
trying to do something with the analysis, and it's 
got all kinds of problems, and now you asked me to 
focus on one little piece of it and say how — what 
should I do there to fix that piece, it — and how 
would it address it, without addressing all the other 
pieces. I mean you just have to — in my view you 
have to start — if the objective is to estimate this 
descriptive quantity, this SAF or something like 
that, within categories; age, sex, insurance type, 
seatbelt use, et cetera, and then estimate that 
quantity descriptively within reach of those 
categories, I would just start again, so I wouldn't 
do any of this the way it's being done here. 

Q. Have you run the model to attempt to correct for 
this error that you identify of making the 
conditional independence assumptions across 
expenditure categories? 

A. It wouldn't be correcting that error. I don't 
know why they're doing it that way anyway. In my 
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1 mind — in my mind it's not a way to address even the 

2 question of this descriptive quantity SAF within each 

3 cell. 

4 Q. Do you consider what they did here in making 

5 conditional independence assumptions across 

6 expenditure categories an error? 

7 A. An error with respect — is this — when someone 

8 makes an assumption and it's — it's probably not 

9 very well supported by the data. That is — is an 

10 error. Now to correct it means to understand what 

11 you're trying to do and to try to do it correctly, 

12 and if this is a piece in a puzzle that doesn't — 

13 the rest of the puzzle doesn't work right, you can't 

14 correct this error. I can — 

15 Q. Have you run the Zeger, et al. model without 

16 making this — 

17 A. No, I don't — 

18 Q. — assumption? And do you know what difference 

19 it would make in Plaintiffs' damages if you were to 

20 run the model without this assumption? 

21 A. I do not know what difference it would make on 

22 the answers that have been produced. 

23 Q. And with respect to this category of, as you put 

24 it, errors that you call reliance on strong 

25 unsubstantiated assumptions, would it be correct to 
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say that you do not know what effect there would be 
on Plaintiffs' damages if you ran the model prepared 
by Zeger, et al. without those assumptions? 

A. I don't even know quite what that means, to do 
it without the assumptions, but I certainly don't 
know, even if you specified that more precisely, what 
difference it would make on the numbers being 
produced as — from running the Zeger model. 

Q. Now, you go on to say that "Neither the 
plausibility of these various assumptions nor the 
sensitivity of the Zeger Report's results to these 
assumptions appears to have been investigated." 

Do you see that? 

A. Yes. 

Q. Have you investigated the plausibility of these 
various assumptions? 

A. No, I have not. 

Q. Have you looked at the sensitivity of the Zeger 
report's results to these assumptions? 

A. No, I have not. 

Q. So you don't know whether these various 
assumptions are plausible or not; right? 

A. That's correct. 

Q. Now let's go on to I think your final category 
there which you identify as "Arbitrary Imputations," 
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and you say that "In some instances, the Zeger Report 
arbitrarily imputes missing values." 

Do you see that? 

A. Yes. 

Q. Have you investigated that problem to determine 
— well, strike that. 

Have you done any imputation in the Zeger model 
to fill in those missing values? 

A. I'll interpret that to mean have I done any 
imputation in the data sets used by the Zeger model? 
And I have not. 

Q. Have you run the Zeger model imputing missing 
values in a way that you do not believe is arbitrary? 
A. No, I have not. If that's Zeger model rather 
than Zeger — data sets underlying the Zeger model, 
it's understood to be imputations in the data sets. 

Q. So you don't know the effect on Plaintiffs' 
damages of running the Zeger, et al. model without 
the imputations that you have identified in this 
particular section; right? 

A. Well I don't think that question I can answer 
because you're saying do it without the imputations. 

I think — if you meant to say doing the imputations 
in a proper way, would that be a fair interpretation 
of that question? 
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Q. Sure. 

A. Yes, I have not. 

Q. All right. So again you wouldn't know what the 
effect would be on Plaintiffs' damages of doing the 
imputations in a proper way; right? 

A. I don't know what the effect would be on the 
point estimates that come out of the model, but 
again, if you do imputations in a way that reflects 
the uncertainty about which value to impute, 
certainly standard errors will get bigger and 
confidence intervals will get broader. 

Q. But at this point you don't know how much larger 
the standard errors would be? 

A. That's correct. 

Q. And you don't know how much larger the 
confidence intervals would be? 

A. That's correct. 

(Plaintiffs' Exhibit 3544 marked for 
identification.) 

Q. Sir, I want to show you what has been marked as 
Exhibit 3544. It appears to be a memorandum to you 
from T.E. Raghunathan. 

A. Correct. 

Q. And it bears a date at the top of July 23, 1997 
and then there's another date, July — I can't tell 
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if that's the 23rd or 25th, but in any case, Mr. 
Raghunathan sent you this memo regarding the 
Minnesota files; right? 

A. Correct. 

Q. And the files that he's referring to are — are 
what? 

A. Let me glance at it again. 

Q. Sure. 

A. Because I haven't looked at this for a while. 

Q. Sure. 

A. It's NMES. I think it's NMES. Is it all NMES 
or does it maybe have some BRFSS in it too? This 
appears to be NMES. I believe it's all NMES. I 
might be wrong, I'd have to look at it more 
carefully, but that's — 

Q. Is this one of the two memos that you identified 
as — 

A. Yes. 

Q. — being from Mr. — 

A. This is the one. 

Q. — from Professor or Dr. Raghunathan? 

A. Yes, this is the first of the two that I was 
thinking of. 

(Plaintiffs' Exhibit 3545 marked for 
identification.) 
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Q. I want to show you what has been marked as 
Exhibit 3545. Can you identify this? 

A. Yes, this is the second of the two memos from 
Raghunathan that I referred to. 

Q. And this was prepared by Dr. Raghunathan? 

A. Right, with my direction. 

Q. Is there a date on — well, I guess there is. 
September 3, 1997; is that right? 

MR. BIERSTEKER: August the 22nd, Tom. 

MR. HAMLIN: Is it? 

MR. BIERSTEKER: Yeah. At least that's the 

fax. 

MR. HAMLIN: Okay. All right. 

A. Yeah, that sounds about right. 

BY MR. HAMLIN: 

Q. I can't read the handwriting at the top very 
well. Can you? 

A. Yes, that's mine. "NMES propensity scores - 
initial." Initial, I guess. Then "Sections ABC, 
each six pages, one page for each sex x age group. 
Vars," that's shorthand for variables, "as used in 
Minnesota." Oh, "accepting imputations". So I said 
just pretend as if all those imputed variables were 
real and just let's see what — what comes out of it, 
because I am — I didn't want to go — go beyond it 
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at that point. 

Q. Now section A is titled Main Effects Model. 

What are you referring to? 

A. I'm referring to — well let me — let me go 
back a second. These are estimates of propensity 
scores in each of the six age by sex cells used by 
Zeger, et al., and in each cell there were a 
collection of background variables that Zeger, et al. 
used in their models to adjust, so I just accepted 
what those background variables were, and the Main 
Effects Model just uses those variables without any 
interactions or higher order terms to estimate 
propensity scores, to see how far apart the smokers 
and nonsmokers are. For example, in this first top 
page, are in the Males 19-34 cell, how far apart the 
smokers and nonsmokers are with respect to just the 
main effects of these variables. Section B — 

Q. Well let me — 

A. Sorry. 

Q. — ask you about — 

A. Sure. 

Q. — section A. Do you know what the background 
variables were? 

A. I don't have them in front of me, but I have a 
list somewhere I made up. I don't — but it's a list 
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that was — I told Raghu just to use those variables 
that were used by Zeger, et al. 

Q. Now did you use those variables to predict who 
the smokers were? 

A. Correct. It's basically who's a — who's a 
smoker and nonsmoker within this cell, "cell" meaning 
males 19-34 . 

Q. Did the variables come from NMES? 

A. Yes, this is all on NMES. 

Q. Did any of the variables come from BRFSS? 

A. I don't believe so. 

Q. It's just the NMES — 

A. I believe — 

Q. — variables? 

A. (Witness nods head.) Accepting the imputations 

that were done by the NMES people and by Zeger, et 
al. just to see what would happen. 

Q. And I take it that the conclusion — or at least 
the initial conclusion was to predict the number of 
smokers and the number of nonsmokers; right? 

A. Not quite. The — the exercise was to predict 
being a smoker or a nonsmoker from the background 
variables. So the probability, each person has a 
probability attached to him of being a smoker and a 
nonsmoker as a function of his background variables. 
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Q. What statistical method did you use to make 
those predictions? 

A. Those problem — calculate those probabilities? 
This was a logistic regression. I'm fairly confident 
that almost the same results would have been obtained 
from a probit regression, because with these sample 
sizes you can't really tell much of a difference 
between the probit and the logit specification. 

Q. Who ran the logistic regression? 

A. Raghunathan did. 

Q. And does he have the — the raw data, if you 
will? 

A. Yes, that — well he has the NMES data that was 
forwarded to me and I forwarded to him. 

Q. But does he have the actual regressions 
themselves? 

A. Oh, the regressions — 

Q. Yes. 

A. — that led to this output. Yes, he does. I 
don't, but he does. 

Q. Do you plan to rely on these results for your 
testimony? 

A. I have relied on them for part of my testimony, 
yes. 

Q. Now are these logistic regressions on a computer 
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1 disk? 

2 A. I would have to check with Raghunathan, but if 

3 they are not, they certainly could be put on. I 

4 think, but — certainly is maybe the wrong word. I 

5 certainly expect that they can be put on a disk. I 

6 don't know why they couldn't be. 

7 MR. HAMLIN: Peter, I think we'd want to 

8 see those. 

9 MR. BIERSTEKER: That's fine. 

10 MR. HAMLIN: Okay. 

11 MR. BIERSTEKER: Not a problem. 

12 MR. HAMLIN: All right. 

13 BY MR. HAMLIN: 

14 Q. Now you have listed number 1 here Males 19-34 so 

15 that's the group that you're dealing with on this 

16 first page; right? 


17 

A. 

Correct. 



18 

Q. 

Then you've got four columns. 

The first 

is 

19 

identified as CHARACTERISTICS, the 

second is 

SMOKER 

20 

the 

third is NONSMOKER, and the fourth is SD 

AWAY. 

21 

What 

does SD AWAY mean? 



22 

A. 

Standard deviations between the smokers 

and 

23 

nonsmokers, how — how far away are 

they? And so 

24 

this 

says that they're .92 standard 

deviations way 


25 from each other. Their means are that far apart. 
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Q. Can you take me through the calculation as to 
how you got to .92? 

A. Sure. What I did — I think this is the way 
this particular one is calculated — get this right 
— you take the — the mean for smokers. 

Q. Which in this case is .517? 

A. 794; right. Subtract the mean for nonsmokers. 

Q. Which is .34726? 

A. Correct. And that's the numerator. Okay. The 
denominator has the following form: There are 
different ways to do it, but I think this is the way 
was done. They all come out to about the same, but 
I'll tell you what I think it is. You take the 
standard deviation for the smokers, .18674, you 
square it, you do the same thing for the standard 
deviation for the nonsmokers, .18455, square it, 
average of those two squared standard deviations, 
that's add the two things you just squared, divide by 
2, take the square root of that, and that's the 
denominator. 

Q. That's the denominator? 

A. Correct. In this case whether you averaged up 
the standard deviations or averaged up the squared 
standard deviations and then took the square roots is 
going to make no difference whatsoever because the 
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two standard deviations are so close. But the — the 
quality that I mention that I think is what's done 
here, which is averaging up the squared ones, is a 
quality that appears in theoretical work as — as 
being more interesting than the simple average, so 
that's why it was done that way, although in practice 
which one you use, whether it's the average of the 
standard deviation, the average of the squared ones 
and take the square root, tends not to make much 
difference. So it's possible I did it the other 
way. Possible Raghunathan did it the other way, but 
I don't think so. I think he did it the way I 
described it. 

Q. So if we go back to the column marked 
CHARACTERISTICS, SAMPLE SIZE refers to what? 

A. I hope it refers to number of smokers and number 
of nonsmokers. 

Q. Now is that the number of smokers and nonsmokers 
which are estimated using the variables? 

A. No, that's just the counts in the cell in NMES, 
males 19 to 34. In the data set that we had that's 
how many smokers there are and that's how many 
nonsmokers there are. 

Q. Now the next line is the MEAN? 

A. Correct. 
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Q. All right. Is that — well let's take the mean 
for the smoker. Is that the mean for the smoker as 
predicted by the variables in NMES that you — that 
you used? 

A. Correct. That's the mean of this quantity 
called the propensity scores. That's the average 
probability of — of being a smoker estimated in this 
— in this group. 

Q. Is that what you would call the propensity 
score? 

A. Well that's the average of the propensity scores 
for the smokers. 

Q. Then if we go to the next row it's identified as 
SD; I take it that's standard deviation? 

A. Correct. That's the standard deviation of these 
estimated propensity scores across the 1453 smokers, 
and the corresponding SD for nonsmokers is the 
standard deviation of the estimated propensity scores 
for the 2017 nonsmokers. 

Q. The next column is marked MINIMUM; what does 
that refer to? 

A. That's the minimum of the propensity scores 
across the 1453 nonsmokers and, similarly, of the 
2017 nonsmoker propensity scores. 

Q. And if we go to the next row, that's marked 10th 
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PERCENTILE; what does that refer to? 

A. If you look at the distribution of 1453 
propensity scores, estimated propensity scores for 
smokers, and you look at the bottom 10 percent, 
what's the cutting — cutting value there, so — I 
guess it's the 145th score from the bottom is .26799 
in the smokers, and .12333 among the — the 2017 
nonsmokers, so that's like the 202nd score from the 
bottom for the nonsmokers. 

Q. And the next row is the 25th PERCENTILE; what 
does that refer to? 

A. 25th percentile refers to the — look at the 
bottom quarter, bottom 25 percent of the people, and 
you find the cut point for smokers and nonsmokers, so 
for example, for nonsmokers that would be the 504th 
person from the bottom, from the lowest scores up, 
504th person has a propensity score of .20960. 

Q. The next row is marked MEDIAN; what does that 
refer to? 

A. That refers to the point at which half the 
values of propensity score are below and half the 
values of these estimated propensity scores are 
above, so it's the 726 — 2727 — 727th value for the 
smokers, the 726 below .52020, and there's 726 
above. That's the way, if I can divide properly in 
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my head the 1453. I was dividing 1453 by 2 and I was 
getting 726 below, one in the middle, 726 above. And 
for nonsmokers it would be 1008 below and 1009th 
would be the middle median guy, and the 1008 above, 
and so the 1009th person from the bottom or from the 
top has a propensity score — an estimated propensity 
score of .32595. 

Q. What does the 75th Percentile refer to? 

A. The 75th percentile refers to the person such 
that 75 percent of the values are below that — that 
point and 25 percent are above for smokers, and for 
nonsmokers correspondingly, so if you counted down 
from the top, for example, for the nonsmokers you'd 
have 504th down from the top, 505th I guess 
approximately, right, the — down from the top, the 
504 is about a quarter of the sample, you'd find that 
value would be .46569. My xerox copy is a little bit 
fuzzy there, but that's approximately what it is. 

Q. Let's go to the next row which is the 90th 
PERCENTILE; what does that stand for? 

A. That refers to the point at which 90 percent of 
the propensity scores are below that value for the 
smokers and, correspondingly, for the nonsmokers or 
equivalent 10 percent are above it, so, for example, 
if you went down from the top score is 145 for the 
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1 smokers for the top ones, you'd find the 145th or 6th 

2 one would be .75877, and similarly, if you went down 

3 202 from the top for the largest propensity score for 

4 the nonsmokers, you would find whatever that number 

5 is, I can't read it, .60523 perhaps. 

6 Q. And then finally there's a row marked MAXIMUM; 

7 what does that refer to? 

8 A. That refers to the largest estimated propensity 

9 score among the smokers, .94946, and refers to the 

10 maximum estimated propensity score among the 

11 nonsmokers, which is .89253. 

12 Q. And underneath the box you have listed a 

13 histogram for nonsmokers. 

14 A. Correct. 

15 Q. You see that? Now can you tell me what your 

16 understanding of a histogram is? 

17 A. My understanding of a histogram I think is being 

18 displayed here is a frequency distribution for the 

19 propensity scores for the nonsmokers on top and the 

20 smokers on the bottom. The — the counts in each of 

21 these cells is on the right-hand side and it — the 

22 number of asterisks go out and are proportional to 

23 the counts in those numbers on the right, and each 

24 pile refers to an interval of these propensity 

25 scores. 
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Q. For example, if we look at the 4th row in that 
histogram for nonsmokers on top — 

A. Yes. 

Q. — we see a number of asterisks and then we go 
over to the right-hand side and we see it looks like 
the number 33. Maybe not. 

A. I think it's 40. 

Q. Yeah, it's 40. It's 40. 

A. Right. 

Q. And what does that 40 represent? 

A. That 40 represents the fact that in the interval 
represented by this region there are 40 people with 
that — with that score. In order to figure out what 
that region is I'd have to count up the number of 
rows and divide to see whether I — see what the 
intervals for these — these are. Let me — 1, 2, 3, 
4, 5, 6, 7, 8 — so that's 20 into — these are about 

l/20th of one, so about 20 percent, .05, so I believe 
these are sort of bins of width .05. 

Q. The number 40 represents the number of people? 

A. The number of people, yes, with a propensity 
score in that — in that bin of width .05. 

Q. Okay. When you say — well, — 

A. I think — 

Q. Let's look at the left-hand column now. 
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A. Okay. 

Q. The bottom you have .025, at the top you have 
. 075? 

A. .975 I think. 

Q. Is that .975? My xerox is bad. 

A. I think that's what it is, yeah, .975. Mine's 

not too — too hot either, but I think that's what it 
is . 

Q. Tell me what that represents? 

A. I'm not absolutely certain, but I believe what 
it represents, let's look at the bottom one first, 
okay, the .025; I think that represents a bin between 
zero and .05 so that .025 is the mid point of the 
bin. The next bin, so the next row, corresponds to a 
bin from .05 to .1, I believe. And the next bin from 
.1 to .15. And going from the top, the top bin 
which is labeled .975 I believe represents the bin of 
propensity score values from .95 to 1.0. 

Q. Now when you say the bin represents propensity 
scores, now what bin are you referring to? 

A. I'm talking about the — think of a histogram 
which has, let's say, approximately 20 rows to it. 
Each row corresponds to a bin or a cup where you put 
people who have scores in that range, so in that top 
bin that's marked .975, which I believe corresponds 
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to between .95 and 1, there are two people whose 
propensity scores when rounded off to from 3 decimal 
places have propensity scores that are between .95 
and 1.0. 

Q. Now let's go to the bottom of the histogram for 
nonsmokers. 

A. Okay. 

Q. There seems to be a — a row at the bottom that 
indicates "may represent up to 5 counts" it looks 
like. 

A. I — 

Q. Is that what that says? 

A. "* (star) may represent up to 5 counts". Yeah, 
that's probably telling you that when you get over 5, 
let's see — 

Q. Yeah, what does that tell us? 

A. That tells us — it's a pictorial representation 
of the number of people in that bin. You get the 
exact number by looking at the right column, but to 
give a pictorial rough impression of how big it is 
there are these asterisks that are running off and 
they're telling you approximately an asterisk is 
about 5. Because the counts are large, it wouldn't 
be a very nice picture to have 225 asterisks going 
out and 217, it would cover many pages, so it's not 
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very helpful for getting a visual picture of what the 
distribution is like, so instead there's this 
coarsening to give you a visual picture of what it's 
like and then the numbers on the right tell you what 
the exact counts are, and if you wanted to know the 
story of what the distribution looks like, if you 
were interested in the median or a particular 
percentile, you can look at that information on the 
top this way. 

Q. So that description then is basically this: 
Asterisk may represent up to five counts? 

A. Oh, yes, I'm sorry, yes, that's right. Each of 
these asterisks is something approximately up to five 
counts. That's right. So let's see if that makes 
sense. 

Q. And the counts would be the actual number of 
people; right, up to five people? 

A. Correct. Which are seen on the right. 

Q. Okay. And this histogram gives us essentially a 

shape of the data for nonsmokers; right? 

A. Shape of the estimated propensity scores for 
nonsmokers. From this model. On this Main Effects 
Model, which is the title of the section. 

Q. So what we're seeing in that first histogram for 
nonsmokers is the shape of the estimated propensity 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

428 

scores for nonsmokers? 

A. Shape of the distribution of estimated 
propensity scores for nonsmokers; correct, on this 
model, that's correct. 

Q. Now if you'd go over to the right-hand side to 
the column marked Boxplot. 

A. Right. 

Q. You see that. There's a boxplot. And if you 
could explain to me the significance of the top box, 
the top line. 

A. Right. I'm not a great fan of boxplots. Some 
people are. It shows the median — the box in the 
middle of it shows where the median is, that's the 
middle line, the 25th percentile, the 75th 
percentile, and the arms on each side sort of reach 
out toward the minimum and maximum values. I 
personally don't really like boxplots. I'd prefer to 
see the — the actual numbers for the percentiles and 
the histogram which gives a shape. Some people like 
them a lot. 

Q. Does the boxplot show how the — the batch of 
data is distributed? 

A. Basically is a summary of that. It summarizes 
the median, the 25th percentile, 75th percentile, the 
minimum and maximum values in a — in a display which 
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I find less informative than the histogram combined 
with the actual percentiles. 

Q. Well looking at the boxplot for the histogram 
for nonsmokers, what would the median be? 

A. The median corresponds to the line right across 
the middle there. 

Q. Right. 

A. So if you lined it up — I don't even know if 
it's done properly, I mean I don't even know if it's 
aligned properly with the histogram — but supposing 
it were aligned properly with the histogram, and then 
you would look over and the median lies somewhere in 
that bin where there are 225 people, if my eyes 
aren't crossing, and so that means half — somewhere 
in there is the median person for smokers is that 
727th person, which probably is correct. 

Q. Actually that would be for nonsmokers. 

A. I'm sorry. Did I say smokers? 

Q. Yes. 

A. I meant to say nonsmokers. I apologize. Just 
proves how confusing boxplots are to me. 

Q. Then if you look at the histogram for smokers if 
you look at the legend down at the bottom I think it 
indicates "Asterisk may represent up to 4 counts"; 
right? 
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1 

A. 

Yes, it does. 



2 

Q. 

So that's — that's different 

from the 

histogram 

3 

for 

nonsmokers; right? 



4 

A. 

Correct. That's because I — 

I assume 

this is 


5 an automatic, so default of whatever software 

6 Raghunathan was using, but there are 500 or a third 

7 more nonsmokers than smokers, so if you use the same 

8 scale for smokers and nonsmokers they wouldn't look 

9 comparable, because that one would be longer to the 

10 right just because they're more people there, so 

11 there's some default in the program that actually 

12 corrects how many counts are represented in each 

13 stars — each star, each asterisk to make it look 

14 visually comparable in terms of the number of 

15 points. So to try to get it, probably it does some 

16 adjustment to get the number of total asterisks to be 

17 the same no matter how many points there are roughly, 

18 if it can be done. 

19 Q. Now this — this histogram for smokers is 

20 essentially a histogram showing the shape of the 

21 estimated propensity scores for smokers in NMES 

22 predicted by your selection of variables; right? 

23 A. Well it's Zeger, et al.'s selection of 

24 variables. It's my main effect model using those 

25 variables for males 19-34. 
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Q. And it shows the estimate — strike that. 

It shows the shape of the estimated propensity 
scores; right? 

A. For smokers, yes. 

Q. Now do you have a separate list of variables 
that you put together which indicate which variables 
were used to do these estimations of smokers and 
nonsmokers? 

A. Yeah, it was the same list of variables for both 
smokers and nonsmokers, and that list of variables is 
the list of variables that Zeger, et al. used in 
their — in their — list of background variables 
that Zeger, et al. used in their models. 

Q. What I'm asking you is whether you actually have 
a piece of paper that lists out those variables? 

A. I may have it with me, I may have left it at 
home in Boston, but I certainly had it at one time 
and it certainly could be produced. I could probably 
get it with a phone call, if you wanted to know 
today. 

Q. Well I'd like you to produce it. I don't need 
it today. 

A. Okay, no problem. 

Q. I would like to get a — 

A. Absolutely. 
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Q. — copy of that? 

A. Absolutely. Absolutely. It should have been on 
here. I apologize. A more complete report would 
have — a report that's for public consumption would 
have had that information on it and it's certainly 
completely reasonable and expected on your part to 
want to know that list. 

Q. What do you conclude from this? 

A. From just the first page or from — 

Q. From the first page. 

A. From the first page. Well the — the first 
thing that I conclude is that the distributions are 
more than 9/lOths of a standard deviation apart. And 
the shapes of them are a little bit different, but 
the most important conclusion is that they're — that 
far apart is quite far apart in an observational 
study. It's not the worst I've ever seen in terms of 
number of standard deviations between groups, but 
it's quite far apart, and it suggests that any 
modeling to adjust for those background variables is 
going to rely to some extent on extrapolation, and 
therefore I'm not going to trust it very much. 

Q. Now is there a table of standard deviations 
which you use as some kind of a guide to determine 
what is a good propensity score and what is not a 
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good propensity score? 

A. It's not a good propensity score and a bad 
propensity score. The propensity score is revealing 
how far apart these distributions are, the smoker an 
nonsmoker distributions, with respect to the 
collection of background variables. The propensity 
score is sort of summarizing this collection of 
background variables and saying with respect to them 
how far apart they are. So with that understanding 
I'll go on to answer the question, which is that in 
my own work and other people's work on observational 
studies typically one standard deviation apart is 
regarded as being kind of an upper bound beyond which 
you're really — you're really in trouble. That's 
not always true, but it's certainly true in most 
settings. That — that number doesn't just come from 
me in my work with Rosenbaum and Thomas and — and in 
applied settings with a variety of situations, but 
also it comes from my Ph.D. thesis advisor who is 
William Cochran who did a lot of work in 
biostatistics and epidemiology, and he had done — 
seen a lot of observational studies and he always 
felt that anything more than a standard deviation was 
large, so in his articles on adjustments for 
background variables in observational studies his 
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tables always ended at one standard deviation, at 
least that's my memory, and typically he would think 
of a quarter of a standard deviation apart or a half 
a standard deviation apart as something that's in the 

— in the region where you can do adjustments in a 
fairly straightforward way for that set of background 
variables, and beyond that it's — you have to watch 
it, so his tables ended at one standard deviation. 
Simulations that have been done also support that 
fact that things get more and more worrisome as you 

— as you go above a half a standard deviation apart 
and move towards three-quarters and one standard 
deviation. So my conclusion, therefore, is that I'm 
kind of worried about any kind of modeling based 
adjustments for the — in — in a situation like 
this . 

Q. Now can you identify an article on your resume 
where you state that an adjustment — or strike that 

— that a standard deviation of one or more between 
propensity scores is too far? 

A. I didn't say too far necessarily. I said it 
would be of great deal of concern. 

Q. I thought you said too far. 

A. Well it's too far to be relied on for — for 
linear adjustments, that — that's true. It's too 
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far just to go ahead and do linear modeling 
adjustments. I — I — I think a half a standard 
deviation is probably too far to rely on that. I 
don't know whether I can find a statement to that 
effect that is as sharply worded as that, but let me 
take — take a fast look if I may. Okay. I'm just 
looking at the article that you handed me to look at. 
Q. Well I had asked you to look at your resume 
which included a number of articles; correct? 

A. Correct. 

Q. But you want to look at a specific article. 

A. Well I have one that — that's specific which is 

the most recent one that's been published on 
propensity scoring methods, and so it's quite 
possible that I can just find something just to quote 
from that, in which case otherwise I would give you a 
list of probably 10 or 15 articles where such a quote 
might appear. So if you'd prefer my going that 
route, I can do that. 

Q. Well actually let — let's go this route first. 
Now — 

A. Okay. 

Q. — what is the Exhibit number that you're 
looking at? 

A. I'm looking at Exhibit 3542. 
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(Witness reviewing document.) 

A. Let's see where we have simulations here. In 
this article I decided really to push the limits of 
something to see what would happen. If you'd turn to 
page 255 there's this thing at the bottom which is 
Table 1, and it is distribution, where it says 
Simulation settings, and there's a quantity called B 
which is in the second row, "Difference in 
populations means B". That's this thing of number of 
standard deviations apart, so in this particular 
simulation I pushed it to one, one and a half because 
I suspected everything was going to fall apart at one 
and a half, and because we had a theory that we had 
done based on approximations I wanted to see how — 

If you only look at conditions when you hope the 
theory works, then you never see what it breaks down 
to because the theory was based on approximations. 
This was a matching setting I think though. Which is 
easier. A matching setting is easier because you 
have a reservoir so you can throw away the tail of 
the distribution. Here you couldn't really do that. 
So the minimal matching ratio I have here is what? 

2. 2 to 1. So that would be a setting where, for 

example, if I wanted to compare smokers to nonsmokers 
and there are twice as many nonsmokers as smokers, so 
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in fact the distributions are far apart, I can ignore 
part of the distribution, because that's twice as 
many, so it's a simpler setting and — and let's see 
if I can find some advice on that. Or is this advice 
mainly about how well the theory works? I don't 
remember. 

(Witness reviewing document.) 

A. The main purpose was relating the theoretical — 
purely analytic theoretical results to simulation 
results, and some data, so let's see if there's any 
comment about it here now. 

Q. Just keep in mind. Professor, that what I've 
asked you is to tell me if you can cite to a page in 
this article which states that a standard deviation 
of 1 or more is a cause of concern? 

A. Well I think maybe half was a cause of concern. 
One or more would be — I guess because we don't do 
the linear adjustments in this article that I'm 
focusing on it's not going to be in there. And it's 
actually where we actually bring in some linear 
regression adjustments is in a — in a current paper, 
but let me go back to the vitae to indicate articles 
where I've done linear — compared linear regression 
and matching kinds of things. Again this is some — 
this is just indicating how far apart they are. 
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1 

Q. 

When you say "this" 

you're again — 

2 

A. 

This — 


3 

Q. 

— you're referring 

to Exhibit — 

4 

A. 

Yes, I am. 


5 

Q. 

— 3545? 


6 

A. 

Yeah. 


7 

Q. 

The histograms for 

the NMES? 

8 

A. 

Yeah. Let me start 

with an older article 


9 in the univariate case this is a problem. So we go 

10 back to 1973. Probably before Peter was born. 

11 MR. BIERSTEKER: I wish. 

12 A. Number 8. "The Use of Matched Sampling and 

13 Regression Adjustment to Remove Bias in Observational 

14 Studies". And then the subsequent one which was 

15 jointly authored with — with Bill Cochran. 

16 Q. Do those deal with propensity scores? 

17 A. No, they don't, they deal with the simpler 

18 setting with just one variable, so therefore the 

19 propensity score is identical to that one variable. 

20 Q. Let me ask you this: In your published 

21 writing — 

22 A. Yes. 

23 Q. — have you written anywhere that a propensity 

24 score or scores with a standard deviation of one or 

25 more is a cause of concern or is too far, as you put 
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it? 

A. Too far to rely on linear adjustments, and I'm 
telling you that this article will say that the use 
of one variable — the propensity score is one 
summarized variable, so if the — if the summarized 
variable, or if you only started with one variable, 
if that variable has more than a standard deviation 
bias between the two groups, the regression 
adjustment is very sensitive and can lead you 
astray. That's what this article is about is it's 
about that topic of you have a variable and it has a 
different distribution in the two groups and you try 
to do the adjustment for that variable using models, 
linear models, and is it reliable or not? As you get 
farther and farther apart it becomes less and less 
reliable. That's what it shows. And I think it is 
— my memory is — this is a while ago, I probably 
wrote this in 19 — I wrote this in 1969, 70, it was 
part of my thesis — is that — that things really 
start falling apart above a half a standard 
deviation, you get problems. It is relevant to 
propensity scores because propensity score is one 
summarized variable, it's one variable. 

Q. And what I'm asking is: Do you know of one of 
your published articles where you have indicated that 
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a propensity score the summarized variable goes too 
far or is of great concern where there is a standard 
deviation between those propensity scores of one or 
more? 

A. I can — I can look at articles where it — it 
may or may not appear with that particular sentence. 
Okay. This is highly relevant. It doesn't get any 
more relevant than this article that I'm referring 
to. So I'm saying that is — that is relevant. I 
will look for other ones. 

Q. And I'd ask you to look for other ones that deal 
specifically with the subject of propensity scores. 

A. This one does that I just referred you to. I'll 
refer you to another one. 

(Witness reviewing document.) 

A. In 1976 an article, number 19 on the vitae, 
that's also published in Biometrics, which is 
"Maximums on Biased Reduction for Fixed Sample 
Size," that's part of the title, and that has how 
much bias reduction you can get by doing matching as 
a function of the standard deviations apart. 

Q. Does that article deal with propensity scores? 

A. It deals with what is the same thing as 
propensity scores called discriminate matching, 
discriminate in fact is the propensity score, one way 
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of estimating it at least, so it wasn't called 
propensity score, but in fact is — is the same as 
the propensity score. 

Q. Can you restrict your review to articles that 
deal specifically with propensity scores. 

A. Okay. That does deal specifically with 
propensity scores, so I'm — I'm addressing it 
there. 

(Witness reviewing document.) 

A. Number 40, 1979, is an article in JASA which 
deals — titled "Using Multivariate Matched Sampling 
and Regression Adjustment to Control Bias in 
Observational Studies". Again one of the techniques 
is using the discriminate, which is the same as the 
propensity score except it wasn't called that there, 
and it shows how regression adjustment — linear 
regression adjustment models are dangerously used, 
this is in a bivariate setting, two — two 
variables. In that case the discriminate is the 
propensity score, estimated propensity score. That 
— my memory of that is things start falling apart 
when you get more than a standard deviation, more 
than a half a standard deviation things start getting 
sensitive to the regression stuff and gets — I don't 
even know if I go beyond one standard deviation 
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because I think it gets bad enough there. 

I'll continue. 

(Witness reviewing document.) 

A. In number 53 in 1980 in Biometrics, there's an 
article, "Bias Reduction Using Mahalanobis' Metric 
Matching". One of the points of comparison to the 
Mahalanobis metric matching is discriminate matching, 
which again is propensity score matching, and so it's 
discussed there I believe as well. 

It's maybe in the propensity score article 
itself. Should I look in that one since it's here? 
Which is a 1983 one. 

Q. Sure. 

A. Okay. Because I couldn't find a statement. 

That's theory. 

(Witness reviewing document.) 

A. You can see in the abstract of that article it 
says "propensity score, which is a generalization of 
discriminate matching" so I — I was not trying to be 
difficult, I was trying to be accurate. So it's — 
it's not the first time that — here is not the first 
time that appeared. 

Matching. Theory, theory, theory. 

Q. Again what I'm looking for is a statement to the 
effect that when you have a standard deviation of one 
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or more between two propensity scores this is a cause 
of concern? 

A. Isn't as precise as that, but I'll — at the 
bottom of page 48 it says reasons for preferring 
doing some matching, a model-based adjustment — 
model-based adjustment is like linear regression 
adjustment — "Model-based adjustment on matched 
samples is usually more robust to departures from the 
assumed form of the underlying model than model-based 
adjustment on random samples, primarily because of 
reduced reliance on the model's extrapolations." In 
a setting like this you have just samples, you're not 
— not doing the — no matching is being done, no 
elimination of the tail of one distribution is being 
done, so it's saying why propensity score stuff us 
more reliable. I'll continue to see if I can find a 
reference to a number of standard deviations. 

(Witness reviewing document.) 

A. You'll see also at the bottom of page 49, this 
is in Section 2.3, "we observed that discriminate 
matching is equivalent to matching on the propensity 
score if the covariates have a multivariate normal 
distribution." And then it refers to some of these 
articles that I was mentioning before. 

Let's see if there's anything here about one 
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standard deviation. Let's see. Table 1 at the top 
of page 51. Let's see. Here's the kind of thing 
that can happen. You see the initial bias along the 
standardized discriminate, you see in that table, 
that's the label of the columns. And it goes .2, .5, 

.75, 1.0. As I said before, that 1.0 came from 
William Cochran's advice that you don't want to get 
into studies where they're that big. Now you see how 
well you can do trying to control bias. These are 
percent reductions in bias using the technique. 
"Assuming bivariate normal covariates with common 
covariate matrix, parallel linear response" — 
(Reporter interruption.) 

A. I'm sorry, I'm just reading from the bottom 
table. I'll stop there. You'll notice that the 
quarter standard deviation apart is correct, the 
percent reductions in bias which ideally would be 100 
percent means you've gotten rid of all the bias due 
to the covariates that you're trying to control. 

With a quarter of a standard deviation between the 
groups you get rid of almost all of it. 

And you move to half a standard deviation, 
depending upon the ratio of the sample sizes, you get 
rid of a lot of it. You move to three-quarters of a 
standard deviation things start getting worse. You 
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move to one, they're substantially worse, even when 
there's a ratio of 2 to 1, especially when there's a 
ratio of 2 to 1 in the reservoir of the control 
group. So there's some advice follows from that that 
indicates that you can — you're getting — you're 
leaving a lot of residual bias if the standard 
deviation is as large as one. I know it's not 
exactly the sentence you were looking for, but this 
is written for a technical audience, which I'm not 
sure — this is a technical journal, so the audience 
of this article would look at this and conclude that 
without the specific sentence. 

Let me see if I can go on. 

(Witness reviewing document.) 

Q. Well are you saying that if the standard 
deviation is one or more you just basically throw 
away the data? 

A. Not necessarily, no. If you — if you're in a 
setting where you have a big reservoir, let's suppose 
in this situation we had these 1453 smokers and let's 
say 10,000 nonsmokers so the ratio of the reservoir, 
the nonsmoker reservoir is 6 or 7 times as large. 

Then the fact that one distribution is far away from 
the other one still may mean they overlap in some 
region, so you can compare them in that — in that 
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region, but as they get farther apart if you leave 
that other tail of distribution in there and rely on 
drawing straight lines, you can really be in 
trouble. This does not — 

Q. Now is there a way that you can correct for that 
lack of overlap? 

A. Well, using matched sampling, using propensity 
scores subclassification creates groups where the 
overlap is much more, and then you — you can try 
doing these modeling adjustments within these 
subclasses created by the propensity scores because 
there's — there's more overlap, but if they overlap 
— if you don't — if you don't worry about that 
overlap and so go blindly head with linear models or 
any — any kind of specific model, you can get badly 
hurt. 

Q. Did you make any corrections for what you 
perceived to be a lack of overlap in the data in the 
Zeger, et al. model? 

A. No, I did not. 

Q. Okay. Do you know what difference making such 
corrections would make on the Plaintiffs' damages? 

A. No, I do not, but that's my — what underlies it 
is my concern that the modeling kinds of adjustments, 
even for the descriptive quantities that's being 
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estimated, I don't trust the — I don't trust the — 
the model-based adjustments. The answer may change, 
it may not change, but if there is — if the models 
aren't really, really good, I mean really accurate, 
then you're going to get — going to get in trouble. 
And as I look through this output I think in all the 
cells and all the things I looked at there was only 
one case where it was less than a standard deviation 
apart, sometime more than a standard deviation, often 
typically more than three-quarters of a standard 
deviation apart, so I — here's 1.2 standard 
deviations apart, and so I don't have any confidence 
in the results. 

Q. And you took no — or you took none of these 
corrective measures and applied them to the Zeger 
model; right? 

A. No, the first — the first step in an analysis 
that attempts to compare even descriptively two 
groups adjusting for background variables is to see 
how far apart they are, and then the next step is to 
try to do some adjustment for it if you an conclude 
that they're close enough so that it makes sense to 
do it. The other alternative is you can just state 
very explicitly, I'm making this heroic assumption 
about the shapes of these models and let the buyer 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

448 

beware. 

Q. Did you attempt to make any adjustments that 
took into account your results in the propensity 
scoring? 

A. No, I did not. 

Q. All right. And so you don't know then what 
results making those adjustments would have on the 
Plaintiffs' damages; right? 

A. No, I do not. 

Q. Now let me ask you this: If we go to page 2 of 
Exhibit 3545, can you read the mean numbers on that 
page? I really — I can't? 

A. Wait a second. 

MR. BIERSTEKER: Well the page with the 
hand-numbered 2 on the top — 

MR. HAMLIN: Yeah, hand-numbered 2. 

MR. BIERSTEKER: — or the hand-numbered 3 
on the bottom? 

MR. HAMLIN: Well, yeah, it's — 

BY MR. HAMLIN: 

Q. It's Males 35 to 64, and I see the mean, but I 
— I think I can read it for smokers. 

A. Okay. 

Q. I can't read it for nonsmokers and I can't read 
the SD AWAY. 
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1 A. Let me try to address that. I got distracted 

2 because I — I found a paragraph in the original 

3 Rosenbaum and Rubin article that was closer to what 

4 you were looking for, but that — if you want me to 


5 

put 

that aside I will. 




6 

Q. 

Yeah. Let's talk about 

this 

Exhibit 3545. 

7 

A. 

3545, okay. 




8 

Q. 

All right. And can you 

read 

— 


9 


MR. BIERSTEKER: Next page. 


10 

Q. 

— the mean? 




11 

A. 

Oh, the next page. The 

mean 

for 

— yes, I think 

12 

I can. The mean for smokers? 




13 

Q. 

Yeah. 




14 

A. 

It's looks like .13488. 

And 

for 

nonsmokers — 

15 


MR. BIERSTEKER: No 

, no 

, that's the 

16 

standard deviation. 




17 

A. 

Oh, I'm sorry. .66138. 

My 

eyes 

jumped. And 

18 

the 

for nonsmokers .54084. 




19 

Q. 

What is the SD Away? 




20 

A. 

.7331. 




21 

Q. 

Now the next page. Males 

65 

Plus. 


22 

A. 

Right. 




23 

Q. 

Can you tell me what the 

mean is 

for the 

24 

nonsmoker? 




25 

A. 

Mean for nonsmokers appears 

to be 

.54225. And 


STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




CONFIDENTIAL 


450 

1 the standard deviation away is .70114. 

2 Q. Let's go to the next page. Females 19 to 34. 

3 Tell me what the mean is for the smoker. 

4 A. Appears to be .48170. SD Away .7971, that's a 

5 .8 standard deviation. 

6 Q. What is the mean for the nonsmoker? 

7 A. .35018, I believe. 

8 MR. BIERSTEKER: Tom, I might observe, I 

9 don't know of you're working from the fax. We faxed 

10 you a copy and we also sent you a hard copy by 

11 overnight. 

12 MR. HAMLIN: Yeah. 

13 MR. BIERSTEKER: You might have the faxed 

14 copy as opposed to the overnighted copy. 

15 THE WITNESS: Yeah, because this one in the 

16 exhibit are pretty clear. 

17 MR. BIERSTEKER: These numbers are actually 

18 pretty legible on the copies you gave us. 

19 MR. HAMLIN: Okay. Good. Good. All 


20 

right 

Great. 




21 

BY MR 

.. HAMLIN: 




22 

Q. 

Can you turn 

to it 

looks like 

it's page 7 at the 

23 

top. 

section B. 




24 

A. 

Yes . 




25 

Q. 

Main Effects 

Plus 

Two factor 

Interaction terms 
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selected Using Stepwise. 

A. Correct. 

Q. Can you tell me what this is? 

A. Okay. Absolutely. We — from section A we did 
the estimation of the propensity score using all the 
variables, which I still owe you a list of, used in 
the Zeger, et al. report as background variables, but 
I didn't use any interactions among those variables. 
Q. What do you mean by interactions? 

A. Products of the variables. For example, — 

Q. Can you give me an example of that? 

A. Sure. Age by education, age times education 
would be predictive of this interaction. Just like 

Q. Did Zeger, et al. do interactions of age times 
education? 

A. I don't recall whether they did or not. 

Q. So this is something that you're doing? 

A. I'm doing it because one of the important 
components of a propensity score analysis in seeing 
how far apart the distributions are is worrying about 
twists in the multivariate distribution that are 
revealed by using product terms, and sometimes powers 
as well. So what Stepwise means is we did a Stepwise 
regression, which means allows all terms to come in, 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




CONFIDENTIAL 


452 

1 all these product terms to come in, and then throws 

2 out the ones that aren't significant one at a time, 

3 so only leaving in I believe significant and I 

4 believe it was 05 level predictors. For the — for 

5 the purpose of just estimating propensity scores, 

6 this Stepwise procedural, although generally I don't 

7 like it, it's okay for an application like this where 

8 you're trying to look at how far apart these two 

9 groups are. 

10 Q. Did you always use the same two factors to 

11 interact? 

12 A. Well we used all — let's suppose there were 10 

13 basic variables that Zeger, et al. used, then we take 

14 all products of the 10 as being possible, so 10, 

15 choose — choose 2. That's 45 possible products that 

16 are in there. It's latent data multiplied properly, 

17 but I think that's right. 

18 Q. Okay. 

19 A. And we kept the main effects in always and then 

20 threw out the variables that were not significant in 

21 the Stepwise fashion. 

22 Q. Now is this another way of calculating 

23 propensity scores? 

24 A. It's another way that you could — that makes 

25 sense. 
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Q. And why would you do it this way? 

A. Because if — if two — if the distribution of 

background variables between smokers and nonsmokers 
differs not just in means but in variances or 
correlations, you'd pick up those differences by 
putting in interaction terms and squared terms. And 
so it's a way of accounting for other differences 
between the groups other than just mean differences 
on these variables. 

Q. Do you have a standard SD Away for this kind of 
a test? 

A. Same, exactly the same thing. It's just that 
more predictor variables went into predicting the 
propensity score but everybody gets a propensity 
score. It's just that it's a — it's a function of 
— of more background variables, so the analysis is 
completely — this analysis, propensity score 
analysis is completely analogous to the one that was 
done before, everybody gets one propensity score, 
it's just that the estimated propensity score now is 
a function of perhaps 15 or 20 predictor variables 
rather than just the 10 let's say main effect ones, 
just the basic, basic ones. 

Q. If we look at the histogram for the males 19 to 
34, again that's a picture of the distribution of the 
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1 propensity scores — 


2 

A. 

Right, 

the estimated — 


3 

Q. 

— for 

this population; 

correct? 

4 

A. 

— for 

this. Exactly. 

The same people as we 


5 were discussing earlier, the same 1453 smokers and 

6 2017 nonsmokers, the only difference is that each 

7 person's estimated propensity score was predicted 

8 from a larger set of variables. 

9 Q. Would you expect in doing propensity scores to 

10 get a histogram for smokers and nonsmokers that's 

11 essentially identical? 

12 A. Would I expect to get ones that are essentially 

13 identical? 

14 Q. Yeah. 

15 A. No. 

16 Q. I mean you'd expect differences; right? 

17 A. That's why I been writing these papers, that's 

18 right, I expect differences and I want to find out 

19 how bad the differences are. That's what the whole 

20 game is. And if you find out the differences are 

21 modest and you can try to do adjustments for them in 

22 various ways, and you try to do the adjustments. As 

23 they get farther and farther apart, that's what all 

24 this literature is about. That's probably one of the 

25 major reasons I have the positions that I do, because 
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I have done a lot of work on the topic. 

Q. All right. So again, you would expect 
differences in the shape of the histograms between 
smokers and nonsmokers? 

A. Correct, and that's why it's so important to 
begin an analysis doing that, looking for that shape, 
rather than just sort of plunging ahead throwing down 
an arbitrary model. That's exactly right. 

Q. Now if we look at the histogram for males 19 to 
34 on this — this page 7, do you see any overlap? 

A. I see some overlap, but not much. These overlap 
in — the fact is it's a one standard deviation 
away. I've got I don't know how many articles doing 
analysis and simulations showing that if you try to 
do linear regression adjustments in a situation where 
they're this far apart and it's not perfectly linear, 
you really are in trouble. Okay. 

Q. All right. 

A. So this is far apart. I don't know — I don't 
know if you want to characterize visually this is far 
apart, but I'm telling you that with respect to the 
kinds of analysis we're talking about this is very 
far apart. 

Q. And you made no adjustments to any of the Zeger 
data to attempt to create more overlap; right? 
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A. Well you can't create more overlap. The overlap 
is what it is. And so the question is have I done 
analyses to try to — beyond these on the data sets 
used in Zeger, et al.? No I have not. 

Q. I mean you haven't made any adjustments to the 
Zeger, et al. data sets? 

A. No, I have not. 

Q. And so you don't know what the results would be 
if you made those adjustments; right? 

A. Correct. I don't know what difference it would 
make in the results if we did something different 
that I would try to do. 

Q. And the adjustments we're talking about are the 
adjustments when you have propensity scores that are 
a standard deviation or more apart? 

A. Or even a half a standard deviation apart. I 
don't know for the Zeger, et al. analyses what 
difference it would have made to try to do the 
adjustment for background variables in a way other 
than the way they did it because I have not tried to 
do that. All I know is that these groups are far 
apart and all the evidence that I'm aware of suggests 
that relying on those kinds of models is a bad idea. 
Q. So I mean you look at this data and it just 
makes you nervous; right? 
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A. It would make me nervous if it were a trivial 
kind of question. It makes me very nervous in a 
substantial issue like this. 

Q. Now can you turn to Section C. It looks like 
it's page 13. 

A. Sure. 

Q. The title here is Main Effects and all two 
Factor Interaction Terms Included in the Model. 

A. Correct. 

Q. Tell me what that means. 

A. Okay. This is the other end of the three 
propensity score estimations in the sense that all 45 
— let's pretend we have 10 variables in the Zeger 
analysis. I have no idea how close that is to being 
right, but just let's accept that for the moment. 

Then if I can multiply properly, there would be 45 of 
these products terms, and so this is a model where we 
estimated the propensity score from 10 plus 45. In 
fact there may also be some squared terms in here, 
the title may be slightly deceptive, because I think 
I requested that Raghu add squared terms right for 
things such as age, I think age squared is in here, 
so that would be — should be an adjustment. In my 
answer to the previous questions about section B as 
well where I was talking about the Stepwise, it 
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probably included squared terms, the age squared as 
well. Okay. 

Q. Okay. 

A. So I didn't mean to — it was just a slip, 
because the title — I was reading the title and 
forgot that I asked him to include, I believe, some 
squared terms. And I don't know whether he really 
did or not though. I — a more complete report would 
have had the list of variables for you to see, and I 
apologize. 

Q. Now — 

A. But all the variables are in here, only some of 
which were selected out by the significant, 
nonsignificant test for section B. So section C 
everything is in there, whether it's significant or 
not. 

Q. All right. So that's the difference between 
section B and section C? 

A. Correct. 

Q. All the variables are in section C, whether 
significant or not? 

A. Correct, all these interaction terms and 
variables are in there; correct. 

Q. And why would you run it that way? 

A. Well that's to, again, look at how far apart 
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these distributions are, both with — not just with 
respect to their means but also their variances and 
their correlations, their general kind of shapes. 

And even if a variable isn't significant it's 
possible it's still important, because it — the 
significance level of a variable in a prediction like 
this is driven by sample size as well as the size of 
the effect, and it's often wise to not be — not drop 
things out just because they're not significant. In 
fact I mentioned earlier that I'm not a big fan of 
Stepwise methods in general for that reason, although 
in this kind of a context I think it makes sense, 
because I'm not relying on — on these coefficients 
as anything more than as ways to get these predicted 
propensity scores, get to these summaries. 

Q. Now if you looked at these propensity scores and 
were asked to make adjustments, what adjustments 
would you make? 

A. Given that they're 1.2, 1.25 almost standard 
deviations apart. With the sample sizes the way they 
are. Let me see. Well, I guess I would probably 
start by trying to form subclasses based on the 
propensity score, probably six, maybe five, five or 
six subclasses of people. Let me see if I can be 
more explicit to tell you what I mean by five or six 
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subclasses of people. This is a very rough example, 
and I haven't thought about how I would do it because 
I find the fact that they're this far apart to be off 
putting to the idea that you can do reliable 
adjustment for background variables even for 
descriptive purposes, but there are roughly what, 

3500 people, smokers and nonsmokers, about 3600. 
Divide that by six for six subclasses, there are 
about 600 people roughly in a subclass. If I tried 
to divide the data set into six equal pieces it would 
be about 600 each. Let's see. If — I would then 
sort of pool these two histograms together to get a 
full distribution of all the propensity scores. This 
is just an example, just an easy thing to do to get 
started, okay, so I hope you're not going to ask me a 
question would you do anything else and is this a 
final analysis and all that. I'm just trying to be 
helpful to indicate what I would do as a starting 
point. And then I would look at — at the cut point 
for the first 600 propensity scores, so I'm pulling 
both histograms together, I'd look at the cut point 
for the first 600, then the 1200th, then the 1800th, 
2400th, and all the way up, so I've got six bins of 
propensity scores. Now within each bin I'm going to 
have some smokers and some nonsmokers. And the 
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1 lowest bin is going to be dominated by nonsmokers, 

2 because if you look at the histograms you can just 

3 see that — let's — let's do just a very rough 

4 calculation so I can try to be helpful with this. If 

5 you look at the bottom three rows in this nonsmoker 

6 histogram there's something less than 600 people 

7 there; right, but there are about 600. I shouldn't 

8 draw on this, should I? 


9 

Q. 

No. 


10 

A. 

Okay. 


11 


MR. BIERSTEKER: 

Unless you want it to be 

12 

memorialized forever. 


13 

A. 

I will put my pen on 

it, okay. So there are 


14 about 600 down there. Okay. Now that cut point, if 

15 these bins are what I think they are, are going by 

16 05, so that's one, two, three, third row up, okay, 

17 third. Look at the other group for smokers three 

18 rows up, one, two, three. In fact, no — no, no, the 

19 second row up is going to be centered, is going to be 

20 — goes to 05, so this is going to go 05 to 1, and 1 

21 to 1.5, so the first five rows are — are up to .15 

22 in propensity scores. You'll notice going up to .15 

23 in propensity scores for the other group, I guess we 

24 do take the bottom three rows as well, and how many 

25 smokers are down there? 20, 53. Three rows, 53. So 
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we've got a distribution where down there you've got 
53 smokers and about 500 and some nonsmokers. That's 
a pretty unbalanced distribution down there, that 
first subclass. So the variability down there is 
being driven by the fact, if you're going to compare 
smokers and nonsmokers down there, it's being driven 
by the very few smokers you have. Now linear model 
adjustment says that's no problem, we just put 
straight lines through everything everywhere and take 
care of the whole problem. The problem is that down 
there you have very little overlap, that's a — 
that's a ratio of 10 to 1. 

Q. This is in males 19 to 35? 

A. Yeah, that's what you asked me to look at. 

Q. Page 13. 

A. Right. Now let's go the other way. What's the 
top subclass look like? The top subclass is going to 
be dominated by smokers. Roughly how far down do we 
go to get 500 or so? A hundred, two hundred — let's 
suppose you go down to point — the first one, two, 
three, four, five, maybe five rows or something for 
— for smokers, this is just very rough, you maybe 
got 500 smokers there. Maybe you go down six, maybe 
that's closer. 400 or 500 smokers. Six. Well we've 
done the other group. And you go down, in the 
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nonsmokers you see maybe you're — you're at a 
hundred people down there. The most — there are 
more nonsmokers than smokers so this calculation 
isn't quite being done right because it's sloppy 
about those kind of differences, but you can see if 
you go down six rows among the nonsmokers you have 
roughly 100 people, if you go six rows down the 
smokers, the same cut point in propensity score, you 
maybe have, what, 300, something like that, and there 
are fewer smokers than nonsmokers, so again you have 
this big imbalance in the distribution. And all the 
work that's been accumulating over — I been working 
on it since the late 60s, so 30 years — says you're 
really in trouble. 

Q. Well let me ask you this: Have you ever done a 
propensity score analysis of an observational study 
and concluded that the propensity scores were too far 
apart for the results to be meaningful? 

A. Yes, I did. 

Q. When? 

A. This is about maybe 8 years ago was a study for 
the Small Business Administration and IRS. Small 
Business Administration was trying to discover 
whether their loans actually produced healthy results 
for companies, that's the purpose, and they wanted to 
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get — do an observational study because they don't 
randomly give away loans, they give them away to some 
needy companies based upon assets and size and years 
in business and all kinds of other things like that, 
and a promise for being a successful company with the 
loan. They wanted to find a control group somewhere 
of equivalent, quote, "equivalent" companies. This 
was going to be done by industry, by industry 
categories, the way the IRS categorized industries, 
so they went to the Statistics and Income Division 
for its data files on tax — tax records which is a 
lot of information on — on companies. The 
statistics and income tax records that they used for 
building research databases are heavily weighted 
towards big corporations, they over-sample big 
corporations because there's lots and lots of little 
corporations that don't matter much at all for tax 
purposes, for income, to our — our federal 
government, but TRW, General Motors, Ford, they make 
a difference. So the IRS sample statistics of income 
sample was heavily skewed towards big firms, whereas 
the Small Business Administration was heavily 
weighted towards small firms because those are the 
kinds of small business firms that try to get loans 
from Small Business Administration. So they tried to 
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do an observational study, we did propensity score 
estimation I believe within each industry, and it was 
a disaster, a debauchery — 

Q. Did you publish those results? 

A. Ah, that's — no, they were internal to IRS. 
There's another example, which is related to — 

Q. Let me just ask you about that first one — 

A. Sure. 

Q. — now. You were involved in that; right? 

A. Yes, I was a consultant on that. 

Q. Do you know of anyone else who has done a 

propensity score analysis and concluded that 
propensity scores were too far apart for the results 
to be meaningful? 

A. Yes. 

Q. Other than yourself? 

A. Yes. 

Q. Who? 

A. It's in this project with Richard Frank at the 
Harvard Medical School right now Healthcare Policy 
where I mentioned before that we were doing 
propensity score methods with three kinds of mental 
health insurance programs where they had previously 
done some model-based stuff. We did the estimation 
of propensity scores among these there — three 
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groups. We found one of the groups was — was very 
far apart from the other two, I think around a 
standard deviation apart. And the question was. 

Well, Richard, do you want to go forward with this 
and try to do something with it making heroic 
assumptions, or do you just want to focus our 
energies on the two groups where there's overlap? 

And he said. Ah, it's silly, they're — they're so 
different. 

Q. So you were involved in that study; right? 

A. But he made the decision. 

Q. But I'm saying you were involved; right? 

A. But you say — what was the question again? 

Q. Well — 

A. I thought it was had you known anyone else that 
did, not was there anyone else who ever talked to me 
and was I involved. I was involved in it, yes. 

Q. Yes. You were definitely involved? 

A. Right. 

Q. Do you know of any published literature or study 
where someone other than yourself and without your 
consultation has done a propensity score analysis and 
concluded that the propensity scores were so far 
apart that the results could not be meaningful? 

A. If you decided that you can't do a study, you 
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1 probably wouldn't publish that. I don't think many 

2 peer-reviewed journals would like to have an article 

3 that says I thought about doing this but I can't do 

4 anything and therefore it's not there. 

5 Q. So you know — so you know of no published 

6 articles? 

7 A. I can try to publish it. It would be remarkable 

8 if there were any. 


9 

Q. 

So as you sit 

here today you 

don't know of any; 

10 

is 

that right? 



11 

A. 

That's right, 

I don't know of 

any. 

12 

Q. 

Okay. And do 

you know of any 

published results 


13 anywhere where someone has used — well, strike 

14 that. 

15 Do you know of any results anywhere where 

16 someone other than yourself and without your 

17 consultation has done a propensity score analysis and 

18 concluded that the propensity scores were so far 

19 apart that the results could not be meaningful? 

20 A. Is that the same question. 

21 MR. BIERSTEKER: I think it's the same 

22 question, Tom. Could we have it read back? 

23 MR. HAMLIN: Sure. 

24 MR. BIERSTEKER: Because I thought it was 

25 the same question. 
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(The record was read by the reporter.) 

A. I think that's the same question you asked 
before. 

MR. BIERSTEKER: No, the difference is— 

Q. Well it is and it isn't. Let me ask it — 

A. I'm not picking up the subtleties of the 

difference. 

Q. Right. As you sit here today do you know of any 
study by anyone other than yourself and without your 
consultation where there was a propensity score 
analysis done and a conclusion reached that the 
propensity scores were so far apart that the results 
could not be meaningful? 

MR. BIERSTEKER: The results could not be? 
MR. HAMLIN: Meaningful. 

A. So the difference in the question is this one 
does not have published in it anywhere; is that 
correct. 

Q. The difference is that you weren't involved in 
any way. 

MR. BIERSTEKER: No, that's not what the 
difference is. 

A. But you dropped "published" in this question as 
well. 

Q. And I also dropped "published". 
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A. Okay. I can't think of any, because most people 
who had been doing it and found that probably would 
have contacted me, just asked the question. Well what 
can you do? Just the way you were asking what can I 
do. 

Q. All right. 

A. So — 

Q. So as you sit here today you can't think of any 
such study? 

A. That's correct. 

THE WITNESS: Would this be an okay time 
for a break? 

MR. HAMLIN: Sure. Actually it would be a 
great time for a break. I just need to review my 
notes. I may be done. 

THE WITNESS: Okay. 

(Recess from 4:02 to 4:06 p.m.) 

BY MR. HAMLIN: 

Q. Now as I understand it. Professor Rubin, if 
you've got a standard deviation of one-half or half a 
standard deviation between propensity scores you 
wouldn't throw that data away; right? 

A. No, not necessarily, no. 

Q. And if you got a standard deviation of 1 between 
propensity scores, you wouldn't throw that data away 

STIREWALT & ASSOCIATES 

P.O. BOX 18188, MINNEAPOLIS, MN 55418 1-800-553-1953 


http://legacy.library.ucsf.6o'u/tid/xjn05a00/pdf.industrydocuments.ucsf.edu/docs/tjhd0001 




1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


CONFIDENTIAL 

470 

either; right? 

A. Certainly not, if there's a bigger group who I 
can draw from the tail from in a sensible way. 

Q. All right. Would you throw the data away if you 
got a standard deviation of 1.5? 

A. I'd be very tempted to unless one group was 
substantially larger than the other. Or could — 
could change the question in some way being asked. 

Q. What about two standard deviations, would you 
then discard the data? 

A. Again depends upon the matching ratio. But I 
did think — I was involved in that, okay. I was 
thinking of an example of a propensity score analysis 
in which — but I was involved even though I didn't 
do it — where people basically decided that it was 
too far away to do anything with that way, but it 
involved — I was involved in it so presumably you'd 
want to hear about it. It involved a graduate 
student at M.I.T., an — an economist named Bob 
Solow. But I was involved in it. 

Q. And what were the standard deviations or 
deviation between the propensity scores? 

A. In that — in that study? 

Q. In that study? 

A. I don't remember, but it was an example where 
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one of the groups was substantially larger than the 
other group, I think, so there was more chance to 
pick from the tail of one. In fact, yeah, — 

Q. Do you know — 

A. I'm sorry. 

Q. — what the standard deviation between the 
propensity scores was in that Small Business 
Administration study? 

A. It — I do remember that it varied by industry, 
and I — for some reason I remember 14 industries, 
that number 14 sort of comes to me, but I don't know 
whether that's right or not. Maybe it should have 
been 24, I don't know. And do I have a memory of 
what they were? I'm hesitating since I'm really not 
sure but I'll try to — I think in some industries 
they were quite substantial, it would be over 2. I 
believe in some industries they were between 1 and 
2. And maybe in some industries they were like maybe 
3 or 4. But again that was a setting where you 
thought you might have had a chance because the IRS 
database, statistics and income database was huge, 
and the number of Small Business Administration firms 
that received loans was much smaller, so you might 
have hoped that despite the fact they're relatively 
far apart, the one distribution had so many more that 
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1 its tail might come across into the other — into the 

2 other group, not like those histograms we're looking 

3 at today with the sample sizes being roughly equal 

4 for smokers and nonsmokers within the cells. 

5 Q. And the histograms you're referring to are in 

6 Exhibit 3545? 

7 A. That's — that's correct. 

8 Q. Okay. All right. Thank you very much, 

9 Professor Rubin. 

10 A. Okay. 

11 (The deposition was in recess at 4:11 p.m.) 
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